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The Effects of Math Pathways and Pitfalls 
on Students’ Mathematics Achievement 

Executive Summary 

This study addressed two sets of questions. First, the evaluation was designed to measure the impact 
of Mathematics Pathways and Pitfalls (MPP) on the mathematics that second-, fourth-, and sixth-grade 
students learn. The specific research questions that were addressed are: (a) What is the impact of MPP on 
students’ knowledge of the mathematics topics addressed, compared to that of students using the regular 
math curriculum? and (b) How equitable is the impact of MPP on students’ mathematics knowledge 
across levels of English language proficiency and entering mathematics ability? To contribute to the 
interpretation of the results, the research also examined the fidelity of lesson implementation as enacted 
within MPP classrooms, compared to the structure and processes that were intended by the curriculum 
designers. Questions that were addressed are: (a) How closely does MPP as enacted follow the structure, 
content, and discourse processes that were intended by the curriculum designers? and (b) How does MPP 
as enacted in classrooms that had greater student math score gains compare with MPP in classrooms with 
lower student gains? 

Study Design 

A cluster-randomized experimental design was implemented in five school districts across the nation 
over a two-year period. In the first year of the study, second-, fourth-, and sixth-grade teachers were 
randomly assigned within their school districts in roughly equal numbers to either an experimental group 
or a control group. The experimental group teachers were taught how to implement MPP during a six- 
hour professional development (PD) session in the summer. Project consultants, trained by project staff, 
conducted the PD. In the first year of the study, experimental group teachers substituted MPP for a 
portion of their regular mathematics curriculum. The control group teachers used their regular 
mathematics curriculum, and received whatever professional development they normally were provided 
during that year. A total of 99 teachers and 1,971 students participated in the first year of the study. 

In the second year of the study, control group teachers received four days of PD and then substituted 
MPP for a portion of their regular mathematics curriculum. The experimental group teachers from the 
first year of the study had the option of teaching MPP again in the second year. These teachers were not 
obligated to do so, however, and because many elected not to participate in the second year of the study, 
results from the first year only are presented in this report. 

Methods 

Teachers completed a background questionnaire, an implementation questionnaire, and an end-of- 
year evaluation questionnaire. The MPP Pitfalls Quiz, which was given at the beginning and end of the 
school year, was the primary instrument used to measure students’ mathematical knowledge. A separate 
MPP Pitfalls Quiz was developed for each grade level, each of which was aligned with MPP for that 
grade level. Items on the MPP Pitfalls Quizzes assess concepts and procedures that are known to cause 
difficulty for students as identified from the research literature and prominent assessments such as the 
NAEP and TIMSS. The tests contain one or more items that relate directly to the content of each lesson, 
and a few additional items that were indirectly related to the lesson content. Most items on these tests 
were multiple-choice format, with one correct answer. At least one of the choices for items in the 
multiple-choice format contained a common misconception that students have with regard to the concept 
being assessed. In addition to the MPP Pitfalls Quizzes, standardized mathematics achievement test score 
data were collected. 
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Results 



Because of the hierarchical structure of the data, with students “nested” in classrooms, multilevel 
models, also known as hierarchical linear models, were used to analyze the MPP and standardized 
achievement test data. These analyses showed that student math performance in MPP classes was higher 
than in non-MPP classes for all three grade levels. With respect to the impact of MPP on students’ 
mathematics knowledge across levels of English language proficiency, we found that for second and 
fourth grades, MPP impacted ELL and non-ELL students equally. The effect size statistics {ESS) for 
second and fourth grade were .43 and .66, respectively. For sixth grade, MPP had a greater treatment 
effect for ELL students {ESS = .74) than non-ELL students {ESS = .28). There were no differences 
between the effectiveness of MPP for students with mathematically stronger versus weaker entering 
knowledge except at fourth grade, where MPP was more effective for children who had higher pretest 
scores. On standardized mathematics tests, for the fourth and sixth-grade students, no statistically 
significant difference was found between the means of control and experimental group students. (No 
analysis was performed for the second-grade students because standardized-test-score data were not 
available.) 

The positive impact of MPP on student mathematics performance, as measured by the MPP Pitfalls 
Quizzes, is consistent with results of an earlier pilot study of MPP materials by Heller, Gordon, 
Paulukonis, and Kaskowitz (2000). Because the current study was based on a more rigorous research 
design (i.e., a cluster randomized design) than the one used in the Heller et al. pilot study, the results of 
the current study can be viewed as even stronger evidence of the effectiveness of the MPP materials. 

With respect to fidelity of lesson implementation, analysis of classroom audio recordings and teacher 
questionnaires revealed that (a) almost all teachers implemented every major component and intended 
discourse process of the lessons; (b) teachers made some minor modifications to the lesson structures — 
namely some steps or prompts were left out more than others, particularly in lower-scoring classes; 

(c) some of the tools for building extended student talk about math, such as the Discussion Builders, are 
spontaneously used by teachers and students, even for lesson segments that are not guided by specific 
prompts in the teaching guide, and during class time on subjects other than math; (d) in classes with 
higher-scoring students, there was more use of Discussion Builders by both teachers and students, were 
asked to explain their thinking less frequently than in lower-scoring classes but more often talked about 
the math among themselves, and gave longer responses about the math. 

Teachers expressed strongly positive opinions about the value of the program, including that their 
students understood the math topics in the lessons better than students in past years, that MPP helped 
most of their students learn the math concepts and prevent pitfalls, and their students really liked MPP. 
Overall, the teachers strongly agreed that they would love to use MPP again next year, and students in 
their schools would benefit greatly if all of the teachers used Math Pathways and Pitfalls. 
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The Effects of Math Pathways and Pitfalls 
on Students’ Mathematics Achievement 



This study was designed to evaluate how the Mathematics Pathways and Pitfalls (MPP) lessons 
impact the mathematics that students learn, and the equity of learning across groups of students at 
different levels of language proficiency. The MPP instructional materials take the unique approach of not 
only fostering correct ways to represent and reason about mathematical concepts, but also explicitly 
calling students' attention to common pitfalls and misconceptions. MPP also provides lesson-specific 
assistance in both the student materials and the teacher's guides for learning how to use mathematical 
vocabulary and symbols, present complete and coherent explanations orally and in writing, and 
participate in mathematical discourse. Students learn to present, expand, justify, and prove or disprove 
mathematical ideas in paired, small-group, and whole-class settings. An important goal of the lessons is 
to help students become careful critics of their own thinking and take a proactive stance toward their own 
learning. 



Math Pathways and Pitfalls 

Math Pathways and Pitfalls (MPP) for K-7 students was developed and field-tested with grants from 
the NSF (ESI 9911374) and Stuart Foundation. The program has broad appeal, especially in the existing 
climate of accountability, since it addresses some of the toughest math concepts and associated learning 
pitfalls culled from the research literature and from national and international assessments. The 
mathematical topics for grades K-3 focus on developing whole number concepts and operations, while 
the topics in grades 4-7 focus on developing rational number concepts. These supplementary materials 
address the need for improving instruction, regardless of the core instructional materials being used. 

MPP consists of video and print materials which include: (a) eight units, one each for grades K-7, in 
English and Spanish, each with 10 to 12 core lessons and follow-up mini lessons for students; 

(b) teaching guides for each lesson as well as each mini lesson; (c) four videos — two professional 
development videos for teachers and two for students — that model how to present and discuss 
mathematical ideas); and (d) Pitfalls Quizzes for each grade to assess math learning. 

Each MPP lesson uses a consistent, easy-to-follow format and includes sections that (a) introduce key 
words and symbols; (b) promote discussion about two excerpts of student dialogue: one that contains a 
correct example of student thinking and another that contains a pitfall in thinking; (c) provide teacher- 
guided and individual practice; and (d) reinforce each concept through follow-up mini lessons, one 
requiring responses to multiple-choice questions and the other eliciting written explanations of a 
mathematical idea. 

Theoretical Framework for Math Pathways and Pitfalls 

An extensive review of the research literature identified fundamental representations and approaches 
to developing mathematical concepts, as well as common misconceptions and conceptual “snags”, which 
we call “pitfalls”. Specifically, the primary-grade lessons on number and operation concepts draw 
primarily on that of Carpenter and Moser (1983), Fuson (1992), Griffin (1998), and Sowder (1992). The 
lessons in the upper elementary grades draw on the rational number research of Behr, Lesh, Post, and 
Silver (1983), Carraher (1996), Moss and Case (1999), Parker and Leinhardt (1995), and Weame and 
Hiebert (1989). 

The framework in Table 1 describes the theoretical foundation underlying the development of MPP. 
The columns identify the critical features of the materials, the theory underlying the feature, and the 
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expected student benefits. In the left-hand margin are the key foci — mathematics, and academic language, 
discourse, and equity — that drove the development of the materials. 

Table 2 provides a synopsis of selected lessons to give examples of the mathematical concepts and 
pitfalls targeted by MPP. 



Table 1 

Theoretical Foundation for Design Features and Expected Student Benefits of Math Pathways and Pitfalls 
Model 

Critical Feature 

of MPP Model Theoretical Basis Expected Student Benefits 
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• Explicit strategies motivate 
students to become careful 
critics of their own thinking, 
justify ideas logically, and 
question the validity of 
ideas. 

• Pitfalls related to important 
mathematical concepts are 
used as a springboard for 
inquiry and learning. 

• Lessons stimulate creative 
solutions to non-routine 
problems and use of a 
variety of representations. 

• Lessons build on prior math 
concepts and connect to 
related concepts within each 
unit and from grade to 
grade. 



Successful students develop 
intentional learning 
strategies for knowledge- 
related goals. Unsuccessful 
students focus on surface 
features (Scardamalia & 
Bereiter, 1983). 

Cognitive dissonance 
stimulates spontaneous 
inquiry and meaning 
construction (Festinger 
1957;Borasi, 1994). 
Inability to solve problems 
with misleading features is 
symptomatic of fundamental 
misunderstandings (Moss & 
Case, 1999). 

A spiral curriculum links 
new and prior learning to 
achieve knowledge breadth 
and depth and facilitate 
extrapolation (Bruner, 1960, 
1966). 



• Students become 
increasingly independent 
mathematics learners, 
elevate the quality of their 
work, and monitor their own 
thinking for pitfalls. 

• Students acquire “habits of 
mind” that incorporate 
inquiry and critical thought. 

• Students gain complex 
understandings that adapt to 
different contexts and are 
resilient to misleading cues. 

• Learning is cumulative, 
generative, and strengthened 
from grade to grade. 
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Lessons model inventive 
student ideas and logical 
reasoning. Discussion 
Builders model ways to 
build on or disagree with an 
idea respectfully. 

Students prove or disprove 
the validity of mathematical 
statements. 

Lessons introduce math 
vocabulary and symbols and 
point out language pitfalls. 
Teaching guides suggest 
ways to make mathematical 
discourse accessible to 
students and achieve broad 
participation. 



• Cognitive apprenticeship 
and scaffolding support the 
new cognitive behaviors and 
patterns of discourse 
(Brown, Collins, Duguid, 
1989; Gibbons, 2002). 

• Knowledge is socially 
constructed, with discourse 
playing a major role in 
developing meaning 
(Vygotsky, 1962; Cobb, 
Wood, & Yackel, 1993). 

• Attending to language and 
status issues enhances 
discourse participation 
(Cohen, 1982; Khisty, 1995; 
Secada, 1992). 



Students build their capacity 
to think inventively and 
reason logically. 

Students are open to 
presenting mathematical 
ideas and examining their 
validity with their peers. 
Students are prepared for the 
discourse expected in 
demanding curricula and 
advanced mathematics. 
Students, regardless of their 
language background or 
social status, increase their 
contributions to 
mathematical discourse. 
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Table 2 

Synopses of Selected Lessons 

Naming Equal Fractions 

Infinite Names for Equivalent Fraction Amounts 

This lesson is an opportunity to help students realize that a region can be divided into an infinite number of 
equal parts and that an infinite number of fractions can name the shaded amount. To find other fraction 
names, Teresa (a fictional student in the lesson) models the idea that you can divide the same region into 
more or fewer equal parts, as long as the ratio of the shaded amount to the whole amount remains the same. 
Visualizing more or fewer subdivisions when naming fraction amounts using any model is a valuable 
strategy that helps children think about equivalent fractions. 

Pitfall: Students count number of parts shaded for the numerator and the number of parts not shaded for the 
denominator (instead of comparing a part to the whole). 



Decimals Are Fractions, Too 

Using Reasoning to Convert Fractions to Decimals 

In this lesson, a fictional student, Tom, models a way for students to use reasoning to find an equivalent 
decimal in tenths for 1/5. First he thought about the whole, or 1, as 10 tenths. He then thought about how 
many tenths would match the area of each of the 5 fifths the rectangle was divided into. He discovered that 
1/5 is equal to 2 tenths and wrote 0.2. If he renamed the whole as 100 hundredths and it was divided into 5 
parts (or fifths), then each part would be 20 hundredths or 0.20. Likewise, if the whole was 1,000 
thousandths, 1/5 would be 200 thousandths or 0.200. 

Pitfall: Students use the digits in the fraction to make 1/5 into .15 instead of the correct equivalent of 0.2. 



Percent Names for Shaded Areas 
Using Reasoning to Name Percents 

In this lesson, Brendon models how to name the percent for a shaded area that is not divided into 100 parts. 
First he thought about the whole area as 100%. Then, he used reasoning to think about the percent for each of 
10 equal parts given that the whole is 100%. So, if 100% were split equally into 10 parts, each part would 
have a value of 10%. So 4 of 10 equal parts would represent 40%. Brendon’s method of first finding the 
amount for one equal part can be used to change any fraction amount into a percentage. A related 
fundamental understanding is that percentage divides a whole amount into hundredths. This means that 1 
hundredth represents 1%, 10 hundredths represents 10%, 125 hundredths represents 125%, and so on. 

Pitfall: Students think that 4 shaded parts (of 10 equal parts) are equal to 4%, even though each part is 10%, 
not 1% of the whole. 



Review of the Research on Academic Language, Discourse, and Equity 

Language is central to all learning since it plays an important role in the way concepts are formed, 
held in memory, and used in reasoning (Pimm, 1987; Vygotsky, 1978). Yet there is considerable evidence 
that most mathematical instruction in the United States is characterized by little verbalization. For 
example, the TIMSS Video Study (Stigler, Gonzales, Kawanaka, Knoll, & Serrano, 1999) revealed that in 
the United States, 78% of the mathematical concepts in lessons were simply stated by the teacher rather 
than developed through explanations or discussion of examples. In a study of bilingual classrooms, 

Khisty (1992) documented that when teachers introduce mathematics concepts they often teach by giving 
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a few typical examples with little or no discussion of the mathematical ideas behind the examples. The 
few verbal explanations that are provided are often ambiguous, incorrect, or inappropriate. Teachers often 
use vocabulary or symbols assuming that they are meaningful to students, even when words or symbols 
have multiple mathematical meanings or have very different meanings from common speech. This 
language-impoverished approach to the instruction of mathematical concepts is not working well for most 
students, but the negative impact is particularly acute for non-native English-speaking students who need 
to learn to navigate the specialized language of mathematics and do so in their second language. 

Given the prevailing mode of recitation-style instruction, it is not surprising that conceptual 
understanding in mathematics, regardless of students' language background, is much weaker than 
procedural fluency (National Research Council, 2001). Even on relatively straightforward mathematics 
problems, students' understanding easily caves into pitfalls. For example, on the National Assessment for 
Educational Progress (NAEP) only 35% of the U.S. 13-year-olds chose the correct response when asked 
for a number between .03 and .04. Similarly, only about 25% of the U.S. sixth graders correctly 

responded 60% when asked to complete the number sentence: .6 = %. The most likely incorrect 

response for this problem is 6%, which is a prevalent pitfall identified in research (Moss & Case, 1999; 
Parker & Leinhardt, 1995). The research literature and national and international assessments provide 
many similar examples, especially in the realm of rational numbers (Behr, Lesh, Post, and Silver, 1983; 
Carraher, 1996; Moss & Case, 1999; Parker & Leinhardt, 1995; and Weame & Fliebert, 1989). What is 
disconcerting is that students don't just make a mistake; their lack of conceptual understanding prohibits 
them from realizing that their incorrect responses do not make sense, even when it is pointed out. This 
feeble conceptual base leaves students unprepared to tackle higher mathematics. 

The research literature in language acquisition and ELL instruction points to specific ways to tailor 
instruction so that understanding of content in English is enhanced. These strategies include (a) providing 
explicit discussion and preview of vocabulary and lesson structure, (b) building on students’ previous 
knowledge, (c) using discourse markers (i.e., "next,” "after"), (d) using visual aids, and (e) helping 
students develop the ability to regulate their own thinking. (Chamot & O’Malley, 1994; Echevarria, 1998; 
Echevarria & Graves, 1998; Gersten, 1996; Short & Echevarria, 1999; Wong-Fillmore, 1982). Typical 
mathematics instruction does not effectively utilize these practices, or if it does, the ideas are often 
misapplied. When teachers try to lower the linguistic complexity of a task, they often also lower the 
cognitive demand of the instruction. This results in watered-down mathematics instruction, which can 
only widen the already large achievement gap. This is just one example of how academic language, 
discourse, and equity are interrelated. 

Academic language has been defined in the literature in terms of vocabulary, syntax, discourse, and 
language functions as they cut across different contexts of use (Butler & Bailey, 2002; Chamot & 
O'Malley, 1994; Cummins, 1980; Solomon & Rhodes, 1995). Discourse plays a central role in developing 
academic language and in promoting equitable learning. For example, Khisty (1995), in a study of 
mathematical language and discourse notes how a simple concept such as talk can either empower 
students or disenfranchise students. She finds that active dialogue plays an important role in giving 
students access to higher cognitive levels of mathematics, and can increase equity in mathematics 
learning. Other researchers note that student-to-student interaction is most effective when students 
actively provide explanations to each other (Webb, 1985, 1989); and when they communicate about, in, 
and with mathematics (Brenner, 1998). Teacher- to-student interaction is most effective, both for 
academic language development and concept development in a content area, when teachers communicate 
with students slightly above their level of competence and mediate interaction so that students have 
opportunities to produce extended stretches of academic discourse (Gibbons, 2002). 

English Language Learners in some schools receive instruction from teachers trained in Specially 
Designed Academic Instruction in English (SDAIE or sheltered instruction) or in bilingual strategies. 
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However, many of these students spend their time in mainstream classes not designed to meet their needs 
(McKeon, 1994). Their teachers have good intentions, but little training in adapting their instruction so 
that the mathematics content remains rigorous while academic language is incrementally developed. The 
Math Pathways and Pitfalls materials offer support to overburdened teachers by embedding the discourse 
and language acquisition strategies identified from the research literature cited above directly into the 
lessons. The intended goal is for these strategies to become part of regular classroom instruction though 
practice with MPP. In addition, the teaching guides provide mathematically robust examples, and explicit 
discussion probes, so that teachers can guide students towards increasingly sophisticated levels of 
mathematics understanding and discipline-specific use of academic language. Sentence stems written on 
posters model appropriate language for students to use as they learn to participate in mathematical 
discourse. 



Pilot Study on the Impact of MPP Instructional Materials 

Initial Pilot Study (1998-99). Prototype materials were pilot-tested with a diverse group of 233 
students whose teachers replaced 1 1 hours of related instruction with project lessons. Utilizing a quasi- 
experimental design with statistical controls, results indicated that students exposed to MPP materials 
exhibited greater gains in rational number knowledge than a similar group of students exposed to regular 
standards-based instructional materials during the same time period (Heller, Gordon, Paulukonis, & 
Kaskowitz, 2000). Impact estimates were pronounced, with an overall effect size estimate of 0.59 
standard deviations. The results were most noticeable for students who performed at low and medium 
levels in mathematics at the beginning of the academic year, with effect sizes ranging from 0.68 to 0.90. 
Consequently, the gap between less and more able students decreased for the project group and increased 
for the comparison group. In addition, scores improved significantly by the posttest among both native 
English speakers and students who were designated by their district as ELL. This was the case although 
all the materials were pilot-tested in English. 

Research Questions 

This study addressed two sets of questions. First, the evaluation was designed to measure the impact 
of the Mathematics Pathways and Pitfalls {MPP) lessons on the mathematics that students learn. The 
questions that were addressed are: 

1 . What is the impact of MPP on students’ knowledge of the mathematics topics addressed, 
compared to that of students using the regular math curriculum? 

2. How equitable is the impact of MPP on students’ mathematics knowledge across levels of 
English language proficiency and entering mathematics ability? 

To contribute to the interpretation of the results, the research also examined the fidelity of lesson 
implementation as enacted within MPP classrooms, compared to the structure and processes that were 
intended by the curriculum designers. Questions that were addressed are: 

3. How closely does MPP as enacted follow the structure, content, and discourse processes that 
were intended by the curriculum designers? 

4. How does MPP as enacted in classrooms that had greater student math score gains compare with 
MPP in classrooms with lower student gains? 
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Methods 



This study was conducted in second-, fourth-, and sixth-grade classrooms in five school districts 
across the country. Altogether, teachers from 40 schools participated in the study. The number of teachers 
per school participating in this study ranged from 1 to 3. On average, there were 1.5, 1.7, and 1.4 teachers 
per school in second, fourth, and sixth grades, respectively. 

As shown in Table 3, an experimental design was implemented over a two-year period. In the spring 
of 2003, teachers were randomly assigned within their school district to either an experimental group or 
control group. Random assignment was done after receiving informed consent from the teachers. 
Randomization was stratified by grade level within each school district. If there was more than one 
teacher from the same school and grade level, these teachers were randomly assigned to either the 
experimental or control group. 



Table 3 



Experimental Design of Two-Year Study With Pre-Post Teacher and Student Assessments 





Random- 

ization 




Year 1 






Year 2 




Summer 

2003 


Fall 

2003 


Spring 

2004 


Summer 

2004 


Fall 

2004 


Spring 

2005 


Teachers 


A: Experimental 


R 


x‘ 


0 


0 




0 


0 


B: Control 


R 




0 


0 


X^ 


0 


0 


Students 


A: Experimental 


NR 




0 X 


0 




0 X 


0 


B: Control 

-^7-1 y-v 1 r> 


NR 




0 


0 




0 X 


0 



X' - One day of professional development 
- Four days of professional development 



In the summer of 2003, the experimental group teachers were taught how to implement MPP during 
one day of professional development (PD). In the first year of the study, experimental group teachers 
substituted seven MPP lessons for a portion of their regular mathematics curriculum. The control group 
teachers used their regular mathematics curriculum, and received whatever professional development they 
normally were provided during that year. 

In the second year of the study, both the Year I control group and Year 1 experimental group teachers 
implemented seven MPP lessons. The previously-control-group teachers received four days of 
professional development on the use of MPP in the classroom, and the experimental group teachers 
received no additional professional development. 

Districts 

Five school districts in California, Missouri, and Arizona served as research sites. Three of these were 
districts in urban or urban fringe communities, one was in a suburban community, and one site served 
several small rural schools spread over a large geographic area. These sites were selected to provide a 
balance of urban, rural, and suburban populations, as well as diversity in the economic, ethnic, and 
language backgrounds of students. 
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A Memorandum of Understanding was negotiated with each district outlining the responsibilities for 
them and the project. Each district had a site coordinator who had the responsibility of recruiting teachers 
to volunteer for the study from their district. They also were responsible for handling the logistics of the 
professional development meetings with teachers, communicating with teachers, and interfacing with 
project staff. They were also charged with interfacing with their district to provide the project with 
student scores from district-administered standardized achievement test. 

Participants 

The goal for the number of elementary classrooms in the study was 100, with about equal numbers of 
teachers in grades 2, 4, and 6. Depending on the size of the district(s), coordinators at each site were 
charged with recruiting between 15 and 40 teachers. A total of 99 teachers participated in the first year of 
the study, and 41 continued to the second year of the study. Each site coordinator recruited teachers from 
his or her district to request voluntary participation in the study. In some districts all of the teachers were 
in elementary schools, while in others, the sixth-grade teachers were in middle schools. If a teacher was in 
middle school, only the data for the first class period in his or her weekly schedule was included. 

The project staff asked coordinators to make every effort to recruit a diverse group of teachers in 
terms of their background, experience, and teaching philosophy. In order to claim that MPP is practical 
and effective for a variety of teachers, it was important that the study be conducted with a variety of 
teachers and classrooms. 

Site coordinators in each district first met with school administrators to get their commitment, then 
met with teachers to solicit volunteers. The project provided site coordinators with a set of presentation 
slides and handouts for both meetings. Recruitment meetings were held in the spring of 2003. The 
purpose of the teacher recruitment meetings was to provide information about the goals and activities of 
the MPP program and the research study. Teachers who volunteered signed consent forms, which 
informed them of their rights and responsibilities as research participants. 

At the recruitment meeting, teachers were informed that they would receive a stipend to participate 
based on the number of hours of project activities they completed and the district hourly rate. In Year I, 
teachers in the project group received a stipend of $350, and teachers in the control (wait-listed) group 
received a stipend of $200. In Year 2, teachers who had been wait-listed the first year received a stipend 
of $400, and teachers who elected to continue from the first-year project group were paid a stipend of 
$250. Teachers who participated in a separate implementation study in the second year received an 
additional $500 stipend. 

In spring 2005, participants were recruited to participate in the implementation study, a descriptive 
study of MPP lesson enactment in the classroom. Program staff invited 1 7 teachers to take part in the 
study, all of whom were Group A teachers who both taught MPP in Year 1 of the project and also 
volunteered to teach MPP in Year 2 of the project. Of those invited, 1 1 teachers agreed to participate. 

Teacher Attrition 

Some Year 1 teachers in both the experimental and control groups chose not to participate in the 
second year of the study. The actual number of participating teachers is presented in Table 4. Continuing 
in the second year of the study was considered optional for teachers in the experimental group, and only a 
subset of the original teachers in that group continued for various reasons not related to the study. For 
example, one of the study coordinators left their district prior to the second year of the study. As a result, 
communication with the teachers in that district was difficult and several teachers did not continue with 
the project during Year 2. 
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Furthermore, when this study began, funding was assured for only one year. Districts and teachers 
were informed that the second year of the study would only be conducted if additional funding were 
procured. For this reason. Year 2 of the project depended on the commitment of the teachers who did not 
know if the study would continue into the second year. While the second year of funding did come 
through, we do not know how many of the Year 1 teachers were lost because of the uncertainty about 
Year 2. Official word of funding for the second year did not come until May 3, 2004. Because of this 
uncertainty, dates for professional development for the second summer were tentative. Teachers from the 
control group may not have reserved time to attend the MPP summer PD since funding was not assured 
and could have dropped from the study as a result. 

An attrition rate can introduce a confound if the teachers who continue in Year 2 differ from those 
who dropped out. We conducted some preliminary analyses to see if there were systematic differences, 
and found evidence that the two groups did differ. Most notably, we found that teachers who dropped out 
tended to have students with lower MPP pretest scores in Year 1 than teachers who continued. Thus, we 
had evidence that teachers were not dropping out in the second year by chance alone, and that the random 
assignment of teachers to groups in the first year would not hold in the second year. For this report, we 
therefore focus on the first year of data for the study. 

Instruments and Data Collection 

As shown in Table 4, in both the 2003-04 and 2004-05 years, participating teachers’ students took 
project-developed pre- and post-tests, and standardized achievement test scores were obtained from the 
district. In both years, teachers completed questionnaires about their relevant background and about their 
implementation of MPP. 



Table 4 

Schedule of Data Collection 



Instrument 


Sample 


2003-04 


2004-05 


Students 


MPP Pitfalls Quiz pretest 


All students 


Sept/Oct 2003 


Sept/Oct 2004 


MPP Pitfalls Quiz posttest 


All students 


May 2004 


May 2005 


Standardized achievement test (as a covariate) 


All students 


Spring 2003 


Spring 2004 


Standardized achievement test (as an outcome 
measure) 


All students 


Spring 2004 


Spring 2005 


Teachers 


Teacher Information Form 


All teachers 


Fall 2003 


Fall 2004 


End-of-year MPP Questionnaire 


All teachers 


July 2004 


July 2005 


Implementation Study Questionnaire - Lessons 5 
and 6 


Sample of 1 1 teachers 


- 


Spring 2005 


Classrooms 


Classroom audio recording - Lesson 6 


Sample of 1 1 teachers 


- 


Spring 2005 



MPP Pitfalls Quizzes. Items on the MPP Pitfalls Quizzes assess mathematics concepts that are known 
to cause difficulty for students as identified from the research literature and prominent assessments such 
as the NAEP and TIMMS. The tests contained one or more items that relate directly to the content of each 
lesson and a few additional transfer items that were indirectly related to the lesson content. 








For both years of the study, all children in the study received an MPP Pitfalls Quiz as a pretest at the 
beginning of the academic year, and the same MPP Pitfalls Quiz as a posttest at the end of the academic 
year. For each grade level, a separate MPP Pitfall Test was developed (see quizzes for grades 2, 4, and 6 
in Appendix A). Each test was developed to match the MPP lessons for that grade level. 

Most items on these tests were multiple-choice format, with one correct answer. At least one of the 
choices for items in the multiple-choice format contained a common misconception that students have 
with regard to the concept being assessed. A few open-ended items were included on the tests, and 
student responses to these open-ended items were scored as either correct or incorrect. For each MPP 
Pitfalls Quiz, we calculated the number of items answered correctly for each student, and then converted 
these raw scores to percentage correct. Thus, a student’s score could range from 0 (0% of items answered 
correctly) to 100 (100% of items answered correctly). On the second-, fourth-, and sixth-grade quizzes, 
there were 18, 17, and 20 items, respectively. 

Mathematics Standardized Achievement Tests. Districts were asked to provide standardized- 
mathematics-test-score data for all students participating in the study. For each student in the study, 
districts were asked to provide end-of-year standardized-test-score data for the student’s previous grade 
level (which served as the covariate) as well as the student’s current grade. For second grade students, 
standardized-test-score data were only obtained for the end of the second grade. (No standardized-test- 
score data were obtained for the end of first grade for these students because first-grade students are not 
typically given standardized tests). For fourth-grade students in the study, standardized-test-score data 
were obtained for the end of third grade and the end of fourth grade, and for sixth-grade students, 
standardized-test-score data were obtained for the end of fifth grade and the end of sixth grade. 

MPP Questionnaires. Teachers were asked to complete a background questionnaire at the beginning 
of each year, providing information about their education, teaching experience, and current teaching 
context as well as teacher and student demographics (see Appendix B). An end-of-year questionnaire was 
given to all teachers in spring of each year (see Appendix C). This questionnaire was designed to obtain 
teachers’ ratings of agreement with statements regarding their use of each component of the MPP 
materials, their own and their students’ responses to the lessons, and their overall evaluation of the 
curriculum. Finally, an implementation questionnaire was administered at the end of the 2004-05 year to a 
subset of 1 1 teachers in grades 2, 4, and 6 Lessons 5 and 6 to obtain more specific information about how 
teachers implemented MPP (see Appendix D). 

Classroom audio recordings. The 1 1 teachers who completed the implementation questionnaires were 
also asked to collect audio recordings of all discourse in their classrooms during Lesson 6 in spring 2005. 
Teachers were each provided with an audio recorder and asked to collect a complete recording of all parts 
of the lesson. Teachers were told that the purpose of asking them to audio tape the lesson was to study 
how the lesson was implemented. The audio recorder was to be placed around the teacher's neck so it 
would clearly record what the teacher was saying. 

Teacher Participation in Professional Development 

Teachers were randomly assigned in roughly equal numbers to either the experimental or control 
group during the spring of 2003. Since the study was being conducted with three grade levels of 
classrooms, separate summer professional development trainings were conducted for each of these grades. 
Thus during the summers of 2003 and 2004, three professional development trainings were held at each 
of three sites, each within driving distance of the districts in the study. In 2004, however, the number of 
teachers at one professional development site was so small that grade levels were combined for parts of 
the professional development that overlapped. 
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Experimental Group 

Year 1: During the summer of 2003, the experimental group attended a one-day, six-hour introduction 
to MPP that was conducted by project consultants, trained by project staff. Most of the professional 
development consultants had used MPP in their classrooms as teachers. During this brief professional 
development training, teachers (a) received an introduction to the goals and purpose of MPP, (b) observed 
a video of a class participating in an MPP lesson, and (c) participated in a short practicum of how to teach 
an MPP lesson. Project staff carefully designed the agenda and activities for this meeting. 

During the school year of 2003-04, the experimental group attended two, two-hour meetings after 
school. One was held during the month of December or January, and the other held in May. The purpose 
of the first meeting was primarily to check in informally with teachers about the lessons they had taught 
so far and have them look through and discuss the next set of lessons they would be teaching. The site 
coordinator conducted these first meetings. The second meeting was a wrap-up. 

Year 2: Teachers in the experimental group were invited, but not obligated, to teach MPP during the 
2004-05 school year. Seventeen teachers chose this option. They were also invited, but not obligated to 
attend the four-day professional development during the summer. Two teachers chose this option. 

Control Group 

Year 1: Teachers in the control group were waitlisted to receive training for the second year of the 
study, and participated only in data collection activities in Year 1 of the study. They did not participate in 
any professional development in the first year. 

Several steps were taken to ensure that being exposed to the MPP materials through the experimental 
group teachers did not contaminate the control group. First, face-to-face meetings were held with both the 
control group teachers and the teachers using project materials. In these meetings, a project representative 
discussed how participation in this project came with a professional obligation to assist the project in 
giving the materials a fair test. Project consultants who led the professional development explained why 
this is important and gave explicit instructions for not sharing any of the materials with anyone else and 
for not looking at the pretests or posttests prior to or following the assessment. To emphasize the 
importance of this request, both teachers using the project materials and the control group teachers were 
asked to sign an affidavit (see Appendix E). 

Year 2: During the summer of 2004-05, teachers in the control group attended four six-hour days of 
professional development. Project consultants who were trained by project staff conducted this 
professional development. At two sites, the professional development was scheduled over a period of four 
consecutive days. The professional development at one site was scheduled for two, two-day sessions. 
Teachers met each day for six hours. 

The content of the professional development training for these teachers consisted of specific activities 
that are part of a course developed for teachers, plus a one-day introduction to MPP. During the first three 
days of the training, teachers used a set of professional development materials that also had guides for the 
professional development instructors. During these three days, specific tasks were used that provided 
practice-based experiences to guide teachers in self-assessment, analysis and reflection, case discussion, 
and discussion of mathematical background and research readings. The final day of the training was 
essentially identical to the one-day training that teachers in the experimental group received in Year 1, 
introducing them to MPP. The mathematical topic of training for teachers matched the topic of the MPP 
lessons they would teach: grade 2 (whole numbers and operations), 4 (fractions), or 6 (percents). 
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Teacher Implementation of MPP 

In the summer of 2003, experimental group teachers received a binder with three MPP lessons as part 
of their summer professional development training. At an after-school meeting held in December or 
January of Year 1 of the study, these teachers received a second binder with four additional MPP lessons. 
In the front of each of these two binders was a suggested schedule for teaching the lessons. Teachers 
recorded in their binders the dates they actually did the lessons. Each MPP lesson consisted of a two- 
period core lesson and two follow-up mini lessons. The entire lesson took approximately two hours of 
instructional time and was taught once per month. Seven lessons were taught over the course of the school 
year for a total of approximately 1 5 hours of instructional time. 

The schedule was the same for both years of the study. As mentioned, only teachers in the 
experimental group taught MPP in 2003-2004. Teachers who were in the control group in the first year of 
the study taught MPP in 2004-2005. MPP was optional for the experimental group teachers in year 2004- 
2005. 



Data Analysis 

The teacher questionnaires were analyzed using basic descriptive statistics. 

For the mathematics achievement data, it is not appropriate to use linear regression because students 
within the same classroom cannot be assumed to be independent of one another. This is because there are 
likely to be many classroom effects and characteristics that the students share in common, for instance the 
teacher. The data are hierarchical, or nested, in that students exist within classrooms. We have data on 
both these levels — on students (e.g., score on previous standardized math test, whether or not the student 
is an English Language Learner), and on their classrooms (e.g., experimental group). (Because there were 
so few teachers per school, with many schools only having one teacher participating, we did not include 
the students’ school as a level in these analyses). Using ordinary regression would yield incorrect standard 
errors. In particular, since treatment varies between classrooms, the standard error of the treatment effect 
estimate and the associated /(-values would be too low (e.g., Snijders & Bosker, 1993). Multilevel models, 
also known as hierarchical linear models (HEM) are designed to analyze relationships among precisely 
these kinds of nested data (Raudenbush & Bryk, 2002). 

The multilevel analyses were performed for each grade level separately. Furthermore, for each grade 
level, the MPP Pitfalls Quiz data were analyzed separately from the standardized mathematics 
achievement test data. Thus, for each grade level, two sets of analyses were performed: one using the 
MPP Pitfalls posttest scores as the outcome variable, and one using the standardized mathematics 
achievement scores obtained at the end of the student’s current grade level as the outcome variable. 

Although multilevel models were used to analyze both the MPP Pitfalls Quiz data and the 
standardized achievement test data, the approach taken to analyze the two achievement measures were not 
exactly the same. Most notably, in the analysis of the MPP Pitfalls Quiz data, we were able to evaluate 
the effect of ELL status on mathematics achievement. For the standardized-test data, we were not able to 
evaluate the effect of ELL, because there was more missing standardized-test data, resulting in smaller 
sample sizes and fewer ELL students in the samples. 

In the next section, the multilevel model approach used to analyze the MPP Pitfalls Quiz data will be 
explained in detail. The multilevel model approach used to analyze the standardized achievement data, 
which was actually a simplified approach of that used for the MPP Pitfalls Quiz data, will then be 
explained in a later section. 
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Multilevel Statistical Analyses of the MPP Pitfalls Quiz Data 

For each grade level, the full multilevel model consisted of four predictor variables: MPP Pitfalls 
pretest scores (Xi), plus three dummy -variable predictors, as follows: 

X2. Experimental group dummy variable. The experimental group variable was coded as ‘0’ if the 
student was in the control group, and ‘ 1’ if the student was in the experimental group. 

English language learner (ELL) dummy variable. The ELL dummy variable was coded as ‘0’ if 
the student was not an English language learner, and ‘ 1 ’ if the student was an English language 
learner. 

lL\, ELL-by-experimental group interaction. In order to find out if the MPP curriculum affected ELL 
and non-ELL students differently, an interaction term was included in the model. This interaction 
term was constructed by multiplying the experimental group comparison dummy variable (X2) by the 
ELL dummy variable (X3). 

Lastly, in order to find out if students’ initial performance level (as measured by the MPP Pitfalls 
pretest) had a differential effect on how they performed in the experimental groups, a pretest by 
experimental group interaction term was also evaluated. (These interactions are sometimes referred to as 
“aptitude-by-treatment” interactions). This pretest by treatment interaction term, labeled X5, was 
constructed by multiplying the pretest score variable (Xi) by the experimental group dummy variable 
(X2). If this pretest by experimental group interaction is statistically significant for a given grade level, we 
have evidence that the MPP curriculum is effecting students differently depending on their initial level of 
performance. All models included a random intercept for classroom. 

Centering the MPP Pitfalls Pretest Data 

To aid in interpreting the multilevel model results, the MPP Pitfalls pretest scores were centered to 
have a mean of zero. For each grade level, this centering was done by subtracting the pretest mean from 
each student’s pretest score. By centering the data, we were able to obtain adjusted posttest means, the 
estimated posttest means for students with mean pretest scores, where the posttest means are adjusted 
based on centered pretest scores. By centering the pretest scores, the adjusted posttest means can be 
interpreted on the scale of the original metric. 

A Description of the Multilevel Analyses for the MPP Pitfalls Quizzes 

For each grade level, several multilevel analyses were performed. For all the analyses, MPP Pitfalls 
posttest scores were used as the outcome variable. 

For the first analysis, four predictor variables were included in the model: 

1. Xi. MPP Pitfalls Quiz pretest scores (centered), 

2. X2. the experimental group dummy-variable, 

3. X3. the ELL-status dummy -variable, and 

4. X4. the ELL by experimental group interaction term. 

If the treatment-by-ELL interaction term (X4) was statistically significant for a given grade level, then 
two simple-effect comparisons were made. First, a comparison was made between ELL students in the 
control group and ELL students in the experimental group, controlling for all other variables in the model. 
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Second, a comparison was made between non-ELL students in the control group and non-ELL students in 
the experimental group, controlling for all other variables in the model. 

If the treatment-by-ELL interaction term (X4) was not statistically significant for a given grade level, 
then the multilevel model was simplified by dropping this interaction term, and re-analyzing the data 
using a model with only three predictor variables, namely (Xi) to (X3) from the above list. 

As a secondary analysis, we were interested in evaluating the pretest by experimental group 
membership interaction term (X5). If the experimental group by ELL interaction term (X2) was 
statistically significant in the previous analysis, then an HEM analysis was performed by adding the 
pretest by group interaction (X5) to the full-model, as follows: 

1. Xi. Pitfalls Quiz pretest scores (centered), 

2. X2. the experimental group dummy variable, 

3. X3. the ELL-status dummy variable, 

4. X4. the ELL by experimental group interaction term, and 

5. X5. the Pretest by experimental group interaction term. 

If the experimental group by ELL interaction term (X4) was not significant for a given grade level, 
then this term was dropped from the model, and the Pretest by experimental group interaction term (X5) 
was evaluated in this simplified model: 

1. Xi. Pitfalls Quiz pretest scores (centered), 

2. X2. the experimental group dummy variable, 

3. X3. the ELL-status dummy variable, 

4. X4. the Pretest by experimental group interaction term. 

Multilevel Software. The statistical package Stata, published by StataCorp (2005), was used for the 
multilevel analyses (Rabe-Hesketh & Skrondal, 2005). For these analyses, parameters were estimated by 
maximum likelihood estimation using the “xtmixed” command. All multilevel analyses are based on 
listwise deletion of missing data. To obtain adjusted means and standard errors, the Stata command 
“adjust” was used. To obtain the simple effect comparisons, the Stata command “lincom” was used. 

Calculating Effect Size Statistics 

If a dummy-variable predictor was statistically significant, then an effect size statistic {ESS) was 
calculated. For each grade level, this ESS was calculated by dividing the estimated regression coefficient 
obtained in the multilevel analysis by the standard deviation of the posttest scores for all subjects in that 
grade level. 



Results 



School, Teacher , and Student Sample Sizes 

Altogether, there were 40 schools that participated in the study. A summary of the number of teachers 
and students in the study is presented in Tables 5 and 6, respectively. For Year 1, there were 32, 38, and 
29 teachers altogether in second-, fourth-, and sixth-grade classes, respectively, and 577, 812, and 582 
students. Only students who had both pretest and posttest MPP test data were included in these analyses. 
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Table 5 



Number of Teachers per Group by District and Grade 





Year 1 


Year 2 


District 


Control group 


Experimental 

group 


Extended pd 
group (control 
group in Year 1) 


Continuing MPP 
teachers 
(experimental 
group in Year 1) 


Grade 2 


1 


5 


5 


3 


3 


2 


3 


3 


1 


1 


3 


1 


2 


0 


2 


4 


5 


4 


0 


0 


5 


2 


2 


1 


0 


Total 


16 


16 


5 


6 


Grade 4 


1 


8 


6 


7 


6 


2 


3 


4 


3 


0 


3 


3 


3 


2 


2 


4 


5 


3 


1 


0 


5 


2 


1 


1 


0 


Total 


21 


17 


14 


8 


Grade 6 


1 


4 


5 


3 


3 


2 


3 


4 


1 


0 


3 


3 


1 


1 


0 


4 


2 


3 


0 


0 


5 


2 


2 


0 


0 


Total 


14 


15 


5 


3 



Table 6 

Number of Students per Group by District and Grade 



District 


Control group 


Experimental 
group (one 
Day of PD) 


Extended PD 
group (control 
group inYear 1) 


Continuing MPP 
teachers 
(experimental 
group in Year 1) 


Grade 2 


1 


108 


101 


68 


66 


2 


53 


44 


17 


13 


3 


9 


40 


- 


34 


4 


84 


62 


- 


- 


5 


38 


38 


18 


- 


Total 


292 


285 


103 


113 



14 











Grade 4 


1 


186 


136 


141 


125 


2 


57 


70 


52 


- 


3 


63 


56 


38 


38 


4 


111 


64 


11 


- 


5 


42 


27 


24 


- 


Total 


459 


353 


266 


163 


Grade 6 


1 


91 


105 


98 


70 


2 


54 


69 


14 


- 


3 


42 


13 


17 


- 


4 


44 


59 


- 


- 


5 


53 


52 


- 


- 


Total 


284 


298 


129 


70 



The number of teachers who identified their schools as urban, suburban, or rural is presented in Table 
7. Most schools were described as “suburban” (56.6% of teachers) or “urban” (27.3% of teachers); 
approximately 15% were rural. 

Table 7 



Number of Teachers Who Identified their Schools as Urban, Suburban, or Rural, 
by Treatment Group 



Setting 




Control 

Group 


Experimental 

Group 


Total 


Urban 


N 


16 


11 


27 




Pet 


31.4% 


22.9% 


27.3% 


Suburban 


N 


26 


30 


56 




Pet 


51.0% 


62.5% 


56.6% 


Rural 


N 


9 


6 


15 




Pet 


17.6% 


12.5% 


15.2% 


Other 


N 


0 


1 


1 




Pet 


0.0% 


2.1% 


1.0% 


Total 


N 


51 


48 


99 



Teacher Demographics and Mathematics Background 

Information on teachers’ gender and ethnicity is presented in Table 8. The majority of teachers in the 
study were women (89.8%), and most teachers identified themselves as “White” (73.1%) or 
“Black/ African American” (18.3%). 
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Table 8 



Teacher Gender and Ethnicity 



Demographie category 




Grade 2 


Grade 4 


Grade 6 


Total 


Cntrl 


Exp 


Cntrl 


Exp 


Cntrl 


Exp 


Teacher Gender 


Male 


Pet 


0.0 


6.3 


20.0 


11.8 


14.3 


6.7 


10.2 




N 


0 


1 


4 


2 


2 


1 


10 


Female 


Pet 


100.0 


93.8 


80.0 


88.2 


85.7 


93.3 


89.8 




N 


16 


15 


16 


15 


12 


14 


88 


Teacher Ethnicity 


White 


Pet 


87.5 


68.8 


68.4 


81.3 


76.9 


53.8 


73.1 




N 


14 


11 


13 


13 


10 


7 


68 


Black/African American 


Pet 


12.5 


18.8 


26.3 


12.5 


7.7 


30.8 


18.3 




N 


2 


3 


5 


2 


1 


4 


17 


Latino/Spanish/Hispanic 


Pet 


0.0 


6.3 


0.0 


0.0 


7.7 


7.7 


3.2 




N 


0 


1 


0 


0 


1 


1 


3 


Asian/Southeast Asian 


Pet 


0.0 


0.0 


5.3 


0.0 


7.7 


7.7 


3.2 




N 


0 


0 


1 


0 


1 


1 


3 


American Indian or 


Pet 


0.0 


0.0 


0.0 


6.3 


0.0 


0.0 


1.1 


Alaskan Native 


N 


0 


0 


0 


1 


0 


0 


1 


Other 


Pet 


0.0 


6.3 


0.0 


0.0 


0.0 


0.0 


1.1 




N 


0 


1 


0 


0 


0 


0 


1 



Descriptive statistics on teachers’ mathematics training are presented in Table 9. Most teachers 
(71.7%) indicated that they had some college mathematics coursework, and 27.3% said they had a 
bachelor’s degree in mathematics or graduate-level mathematics coursework. Nearly half of the teachers 
(42.4%) had between 3 and 6 days of mathematics PD in the previous xx years, and close to one-quarter 
of the teachers (23.2%) had 7 days or more of mathematics PD. 

Table 9 

Teacher Mathematics Education and Training 



Grade 2 Grade 4 Grade 6 Total 



Math background 




Cntrl 


Exp 


Cntrl 


Exp 


Cntrl 


Exp 




Formal math education 


High school math courses 


Pet 


0.0 


0.0 


0.0 


0.0 


0.0 


6.7 


1.0 




N 


0 


0 


0 


0 


0 


1 


1 


Some college math courses 


Pet 


68.8 


100.0 


70.0 


64.7 


78.6 


46.7 


71.4 




N 


11 


16 


14 


11 


11 


7 


70 


BA or BS in Math 


Pet 


18.8 


0.0 


10.0 


17.6 


0.0 


20.0 


11.2 




N 


3 


0 


2 


3 


0 


3 


11 


Graduate level coursework 


Pet 


12.5 


0.0 


20.0 


17.6 


21.4 


26.7 


16.3 


in math 


N 


2 


0 


4 


3 


3 


4 


16 



16 




Math professional development 



None 


Pet 


6.3 


12.5 


10.0 


11.8 


0.0 


13.3 


9.2 




N 


1 


2 


2 


2 


0 


2 


9 


Up to 2 days 


Pet 


25.0 


31.3 


30.0 


35.3 


28.6 


0.0 


25.5 




N 


4 


5 


6 


6 


4 


0 


25 


3 to 6 days 


Pet 


56.3 


50.0 


35.0 


29.4 


35.7 


46.7 


41.8 




N 


9 


8 


7 


5 


5 


7 


41 


7 days or more 


Pet 


12.5 


6.3 


25.0 


23.5 


35.7 


40.0 


23.5 




N 


2 


1 


5 


4 


5 


6 


23 



Additional information on teacher’s mathematics background and teacher training is presented for 
quantitative variables in Table 10. Because many of these variables were skewed, we will summarize 
these statistics using medians instead of means. The typical teacher in this study had 7 years of prior 
teaching experience altogether, and had taught mathematics at any grade level for roughly the same 
amount of time. Furthermore, the typical teacher in this study had been teaching mathematics at his or her 
current grade level for 3 years. Lastly, the typical teacher reported teaching 5 mathematics classes per 
week. The number of mathematics classes taught per week varied according to grade level. 

Table 10 



Teacher Mathematics Teaching Experience 



Teaching experience 




Grade 2 
Cntrl Exp 


Grade 4 
Cntrl Exp 


Grade 6 
Cntrl Exp 


Total 


Years taught prior to this one 


Mean 


6.94 


8.19 


11.38 


10.29 


8.18 


11.87 


9.58 




Median 


8.00 


6.00 


8.00 


4.00 


6.50 


13.00 


7.00 




SD 


3.97 


5.42 


9.80 


9.60 


7.34 


8.52 


7.90 




N 


16 


16 


21 


17 


14 


15 


99 


Years taught math at any grade level 


Mean 


7.31 


8.00 


11.10 


9.59 


8.32 


9.67 


9.12 




Median 


7.50 


6.00 


8.00 


4.00 


6.50 


6.00 


7.00 




SD 


3.63 


5.33 


9.12 


8.54 


7.05 


8.85 


7.40 




N 


16 


16 


21 


17 


14 


15 


99 


Years taught math at current grade level 


Mean 


4.80 


3.78 


6.65 


3.35 


4.86 


5.73 


4.91 




Median 


3.00 


3.00 


3.50 


3.00 


3.00 


4.00 


3.00 




SD 


3.82 


3.59 


7.24 


2.15 


5.74 


5.23 


5.01 




N 


16 


16 


21 


17 


14 


15 


99 


Number of math classes taught per week 


Mean 


4.87 


5.67 


5.00 


12.80 


8.86 


7.73 


7.34 




Median 


5.00 


5.00 


5.00 


5.00 


5.00 


5.00 


5.00 




SD 


.35 


2.58 


.00 


28.30 


7.45 


7.22 


12.04 




N 


16 


16 


21 


17 


14 


15 


99 



Student Demographics 

Information on students’ ELL status and gender are presented in Tables 1 1 and 12. The percent of 
ELL students in this study was 17.8%, 18.3%, and 16.6%, for second-, fourth-, and sixth-grade students, 
respectively. In terms of gender, roughly half of the students were boys, and half were girls, as would be 
expected. 



[MPP NSF Final Report 070829.doc] 



17 of 43 



Printed 8/29/07 







Table 11 



Student ELL Status by Treatment Group 



ELL Status 




Control 


Experimental 


Total 


Grade 2 


ELL 


Pet 


14.1% 


21.8% 


17.8% 


Not ELL 


Pet 


85.9% 


78.2% 


82.2% 


Total 


N 


284 


271 


555 


Grade 4 


ELL 


Pet 


16.7% 


20.6% 


18.3% 


Not ELL 


Pet 


83.3% 


79.4% 


81.7% 


Total 


N 


449 


306 


755 


Grade 6 


ELL 


Pet 


11.8% 


21.4% 


16.6% 


Not ELL 


Pet 


88.2% 


78.6% 


83.4% 


Total 


N 


279 


276 


555 



Table 12 



Student Gender by Treatment Group 



Gender 




Control Experimental 


Total 


Grade 2 


Boy 


Pet 


51.7% 


49.5% 


50.6% 


Girl 


Pet 


48.3% 


50.5% 


49.4% 


Total 


N 


292 


285 


577 


Grade 4 


Boy 


Pet 


53.7% 


48.5% 


51.5% 


Girl 


Pet 


46.3% 


51.5% 


48.5% 


Total 


N 


454 


324 


778 


Grade 6 


Boy 


Pet 


49.3% 


51.3% 


50.3% 


Girl 


Pet 


50.7% 


48.7% 


49.7% 


Total 


N 


280 


279 


559 



Information on students’ ethnicity is presented in Table 13. Across grade levels, approximately 40% 
of the students were European American. There were roughly equal percentages of African American and 
Latino students in each grade. In the second grade, the percentage of African American and Latino 
students was 30.3% and 16.5% respectively; in the fourth grade, the percentage of African American and 
Latino students was 26.5% and 19.7% respectively; and in the sixth grade, the percentage of African 
American and Latino students was 24.6% and 26.6% respectively. 
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Table 13 



Student Ethnicity by Treatment Group 



Ethnicity 




Control 


Experimental 


Total 


Grade 2 


White 


Pet 


36.3% 


37.9% 


37.1% 


Black/African American 


Pet 


35.3% 


25.3% 


30.3% 


Asian/Southeast Asian 


Pet 


5.8% 


7.0% 


6.4% 


Latino/ Spanish-Origin/ Hispanic 


Pet 


16.1% 


16.8% 


16.5% 


Native American 


Pet 


3.1% 


6.3% 


4.7% 


All other responses 


Pet 


3.3% 


6.7% 


5.0% 


Group Total 


N 


292 


285 


577 


Grade 4 


White 


Pet 


35.3% 


45.5% 


39.6% 


Black/African American 


Pet 


31.6% 


19.4% 


26.5% 


Asian/Southeast Asian 


Pet 


3.5% 


1.8% 


2.8% 


Latino/Spanish-Origin/ Hispanic 


Pet 


20.5% 


18.5% 


19.7% 


Native American 


Pet 


4.9% 


11.1% 


7.5% 


All other responses 


Pet 


4.2% 


3.7% 


4.1% 


Group Total 


N 


453 


325 


778 


Grade 6 


White 


Pet 


42.5% 


43.0% 


42.7% 


Black/African American 


Pet 


20.4% 


28.5% 


24.6% 


Asian/Southeast Asian 


Pet 


4.6% 


3.7% 


4.2% 


Latino/ Spanish-Origin/Hispanic 


Pet 


19.6% 


21.5% 


20.6% 


Native American 


Pet 


4.6% 


0.0% 


2.2% 


All other responses 


Pet 


8.3% 


3.2% 


5.8% 


Group Total 


N 


280 


298 


578 



Analysis of the MPP Pitfalls Quizzes 

Measurement Analyses of the MPP Pitfalls Quizzes 

A summary of basic measurement analyses for the MPP achievement tests is presented in Table 14. 
Cronbach’s alpha — a measure of internal consistency — was roughly equal to .80 for most of the tests. 

(The test that was the exception in terms of these high reliability coefficients was the fourth-grade pretest, 
where the reliability coefficient was .42. This was probably due to the overall difficulty of this test, where 
the average item difficulty was .21). 

Multilevel Analyses of the MPP Pitfalls Quizzes 

Descriptive statistics fox MPP test scores are presented in Tables 15 and 16. The statistics in Table 14 
indicate that for each grade level, the mean change from pretest to posttest was greater in the experimental 
group than in the control group. 
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Table 14 

MPP Pitfalls Test Reliability Analysis (Year 1 and Year 2 Data Combined) 



Measure 


Grade 2 


Grade 4 


Grade 6 


Pretest 


Number of items 


18 


17 


20 


Average item difficulty 


.40 


.21 


.41 


Cronbach’s alpha 


.82 


.42 


.78 


Number of subjects 


794 


1241 


781 


Posttest 


Number of items 


18 


17 


20 


Average item difficulty 


.67 


.40 


.54 


Cronbach’s alpha 


.85 


.80 


.82 


Number of subjects 


794 


1241 


781 



Table 15 

MPP Test Scores by Treatment Group and Grade 



Treatment Group 


N 




Pretest 


Posttest 


Change 


Grade 2 


Control 


284 


Mean 


41.10 


62.34 


21.24 






SD 


21.00 


22.44 


19.91 


Experimental 


271 


Mean 


41.06 


71.67 


30.61 






SD 


22.10 


23.01 


20.09 


Total 


555 


Mean 


41.08 


66.90 


25.82 






SD 


21.53 


23.18 


20.52 


Grade 4 


Control 


449 


Mean 


20.33 


31.80 


11.46 






SD 


10.22 


15.31 


15.16 


Experimental 


306 


Mean 


21.76 


45.92 


24.16 






SD 


10.71 


22.38 


20.53 


Total 


755 


Mean 


20.91 


37.52 


16.61 






SD 


10.43 


19.75 


18.60 


Grade 6 


Control 


279 


Mean 


39.32 


49.14 


9.82 






SD 


20.80 


22.08 


16.68 


Experimental 


276 


Mean 


41.12 


56.99 


15.87 






SD 


19.67 


22.44 


16.17 


Total 


555 


Mean 


40.22 


53.05 


12.83 






SD 


20.25 


22.58 


16.69 



Note. This table only includes data for students whose district-identified 
English language proficiency status was available. 
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Table 16 

MPP Test Scores by Treatment Group and ELL Status and Grade 



Treatment Group 


ELL? 


N 




Pretest 


Posttest 


Change 


Grade 2 


Control 


Yes 


40 


Mean 


37.36 


59.44 


22.08 








SD 


17.02 


20.64 


18.55 




No 


244 


Mean 


41.71 


62.82 


21.11 








SD 


21.55 


22.73 


20.15 


Experimental 


Yes 


59 


Mean 


27.97 


65.63 


37.66 








SD 


18.05 


22.64 


19.66 




No 


212 


Mean 


44.71 


73.35 


28.64 








SD 


21.78 


22.88 


19.81 


Grade 4 


Control 


Yes 


75 


Mean 


18.20 


29.49 


11.29 








SD 


9.58 


13.76 


13.50 




No 


374 


Mean 


20.76 


32.26 


11.50 








SD 


10.30 


15.58 


15.48 


Experimental 


Yes 


63 


Mean 


20.73 


41.36 


20.63 








SD 


10.13 


17.71 


18.72 




No 


243 


Mean 


22.03 


47.11 


25.08 








SD 


10.86 


23.32 


20.91 


Grade 6 


Control 


Yes 


33 


Mean 


34.70 


40.30 


5.61 








SD 


18.15 


19.56 


14.88 




No 


246 


Mean 


39.94 


50.33 


10.39 








SD 


21.09 


22.16 


16.86 


Experimental 


Yes 


59 


Mean 


36.19 


55.59 


19.41 








SD 


19.86 


23.06 


18.17 




No 


217 


Mean 


42.47 


57.37 


14.91 








SD 


19.44 


22.30 


15.49 



Multilevel analysis results are presented in Tables 17 and 18. This table contains adjusted posttest 
means, which can be helpful in interpreting the multilevel analysis results. For the analysis where the 
experimental group by ELL interaction term was statistically significant, these adjusted means were 
obtained from the model with the significant interaction term. For the analyses where the experimental 
group by ELL interaction term was not statistically significant, these adjusted means were obtained from 
the simplified model, which did not include the interaction term. The standard errors are based on the 
multilevel analyses. For making statements about differences between the experimental and control 
group, we reported estimated regression coefficients, which correspond to the difference between adjusted 
means, and z-tests based on them. 



[MPP NSF Final Report 070829.doc] 



21 of 43 



Printed 8/29/07 




Table 17 

Multilevel Model Results Using MPP Pitfall Tests as the Outcome Variable 



Fixed Effects 




b 


SEb 


Z 


p>|z| 


Grade 2 


Intercept (Average Teacher Posttest Achievement) 




61.12 


2.29 


26.68 


.000 


Pretest 




0.57 


0.04 


15.17 


.000 


Treatment Group (Experimental vs. Control) 




10.14 


3.24 


3.13 


.002 


ELL Status 




-1.68 


2.63 


-0.64 


.524 


Treatment x ELL Status 




- 


- 


- 


- 


Grade 4 


Intercept (Average Teacher Posttest Achievement) 




32.29 


2.02 


15.98 


.000 


Pretest 




0.56 


0.06 


9.95 


.000 


Treatment Group (Experimental vs. Control) 




13.10 


3.09 


4.25 


.000 


ELL Status 




-2.02 


2.09 


-0.96 


.335 


Treatment x ELL Status 




- 


- 


- 


- 


Grade 6 


Intercept (Average Teacher Posttest Achievement) 




49.84 


2.02 


24.70 


.000 


Pretest 




0.74 


0.04 


20.84 


.000 


Treatment Group (Experimental vs. Control) 




6.29 


2.87 


2.19 


.029 


ELL Status 




-8.57 


3.27 


-2.63 


.009 


Treatment x ELL Status 




10.20 


4.38 


2.33 


.020 



Random Effects 


Grade 2 


Intercept 

(Variance between teachers = SD (constant)) 


8.07 


Level 1 

(Variance within teachers = SD (residual)) 


16.55 


Intraclass correlation 


0.33 


Grade 4 


Intercept 

(Variance between teachers = SD (constant)) 


8.48 


Level 1 

(Variance within teachers = SD (residual)) 


15.07 


Intraclass correlation 


0.36 


Grade 6 


Intercept 

(Variance between teachers = SD (constant)) 


6.59 


Level 1 

(Variance within teachers = SD (residual)) 


14.39 


Intraclass correlation 


0.31 



Note. The intraclass correlation coefficient was obtained by dividing the 
variance between teachers by the sum of the variance between teachers and 
variance within teachers. 
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Table 18 

Multilevel Analysis: Adjusted Mean Posttest Scores of the Control Group versus Experimental Group by ELL Status 





Cntrl 


Exp 


/Vof 


/Vof 


Treatment by 

ELL Interaction Treatment Effect 


ELL Effect 


Pretest by 
Treatment 


ELL? 


Group 


Group 


Tchrs 


Students 


Sig? 


Sig? 


Sig? 


Sig? 








31 


Grade 2 

555 


No 


Yes 


No 


No 



Yes Adj. Mean 59.44 69.58 

SE (3.17) (3.11) 

No Adj. Mean 61.12 71.26 

SE (2.29) (2.38) 

36 

Yes Adj. Mean 30.28 43.38 

SE (2.63) (2.87) 



No Adj. Mean 32.29 45.40 

SE (2.02) (2.40) 











29 


Grade 6 

555 


Yes 


N/A 


N/A 


No 


Yes 


Adj. Mean 


41.27 


57.76 
















SE 


(3.46) 


(3.04) 
















No 


Adj. Mean 


49.84 


56.13 
















SE 


(2.02) 


(2.04) 















Grade 4 

755 No Yes No Yes 
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For Grades 2 and 4, no statistically significant treatment-by-ELL interactions were found. For both 
Grades 2 and 4, a statistically significant main effect for treatment was found — the adjusted posttest mean 
for students in the experimental group was higher than the adjusted posttest mean for students in the 
control group (estimated regression coefficient b = 10.14, z = 3.13,/) = .002, ESS = 0.43 for Grade 2; and 
b = 13.10, z = 4.25, p < .001, ESS = 0.66 for Grade 4). For second grade students, there was a 10.14 
difference between the experimental and control groups adjusted posttest means. This difference can be 
thought of as the value added by being in the experimental group, after controlling for pretest scores and 
ELL status. For fourth grade, there was a 13.10 difference between the experimental and control groups 
adjusted posttest means. Finally, for sixth grade, there was a statistically significant treatment-by-ELL 
interaction {b = 10.20, z = 2.33,/) = .020). Because the interaction term was statistically significant, 
simple effects were analyzed. For non-ELL students, a statistically significant difference was found 
between the experimental and control groups {b = 6.29, z = 2.19,/) = .029, ESS = 0.28). For ELL 
students, a statistically significant difference was also found between the two groups {b = 16.49, z = 3.58, 
p < .001, ESS = 0.74). Thus, the value added for non-ELL students by being in the experimental group 
was smaller (6.29) than the value added by being in the experimental group for ELL students (16.49), 
after controlling for the other variables in the model. 

To summarize, statistically significant differences favoring the experimental group were found for all 
three grade levels. For second and fourth grade, no statistically significant treatment-by-ELL interaction 
was found. The effect size statistics for these grade levels were 0.43 and .061, respectively, favoring the 
experimental group. For sixth grade, a statistically significant treatment-by-ELL interaction was found. 
The difference between the experimental and control groups was statistically significant for both non- 
ELL and ELL students (effect size statistics equal 0.28 and 0.74, respectively), so that the value added by 
being in the experimental group was higher for ELL than non-ELL students. 

Analyses of the pretest-by-treatment interactions revealed a statistically significant difference in 
Grade 4 {b = .239, z = 2.09, p - .036) only. This finding indicates that the MPP intervention was more 
effective for fourth-grade children who had higher pretest scores than for children who initially had lower 
pretest scores. 



Analysis of the Standardized Mathematics Achievement Tests 

The MPP Pitfalls Quizzes were constructed to directly assess the impact of MPP. As such, the 
analysis of the MPP Pitfalls Quiz data is of primary interest. Standardized mathematics achievement data 
was also collected and analyzed, to see if a more global effect of MPP could be seen. All districts were 
asked to provide standardized mathematics achievement test score data for all grade levels. Three of the 
five districts provided these data for all three grade levels (see Table 19). 

As shown in Table 19, the districts varied in terms of which standardized test they used. Three 
districts used the Stanford Achievement Test, 9"' Edition (SAT-9), one district used the Missouri 
Assessment Program Test (MAP), and one district used both the Terra Nova test (TN) and MAP. The 
districts also varied in terms of the metric they used to report the test scores. Some districts provided 
national percentile ranks (NPR), others provided scaled scores (SS), and others provided normal curve 
equivalents (NCE). 
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Table 19 



Name of Standardized Test and Metric Used by District 







Grade 2 


Grade 4 


Grade 6 


District 




Pretest 


Posttest 


Pretest 


Posttest 


Pretest 


Posttest 


1 


Provided 

Data? 


No 


Yes 


Yes 


Yes 


Yes 


Yes 




Name of 
Test 


- 


SAT-9 


SAT-9 


SAT-9 


SAT-9 


SAT-9 




Metric 




NPR 


NPR 


NPR 


NPR 


NPR 


2 


Provided 

Data? 


No 


No 


No 


Yes 


Yes 


Yes 




Name of 
Test 


- 


- 


- 


MAP 


MAP 


TN 




Metric 








SS 


SS 


NCE 


3 


Provided 

Data? 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 




Name of 
Test 


SAT-9 


SAT-9 


SAT-9 


SAT-9 


SAT-9 


SAT-9 




Metric 


NCE 


NCE 


NCE 


NCE 


NCE 


NCE 


4 


Provided 

Data? 


No 


No 


No 


Yes 


No 


Yes 




Name of 
Test 


- 


MAP 


- 


MAP 


- 


MAP 




Metric 




SS 




SS 




SS 


5 


Provided 

Data? 


No 


Yes 


Yes 


Yes 


Yes 


Yes 




Name of 
Test 


- 


SAT-9 


SAT-9 


SAT-9 


SAT-9 


SAT-9 




Metric 




SS 


SS 


SS 


SS 


SS 



Note. “No” indicates that the school district did not provide standardized test score data 
for a given grade level, and “Yes” indicates that the district did provide standardized test 
score data. 



For these analyses, only fourth-grade and sixth-grade data were analyzed. For the fourth-grade 
analysis, the end-of- fourth-grade standardized total math score served as the posttest variable, and the 
end-of-third-grade standardized total math score served as the pretest. Likewise, for the sixth-grade 
analysis, the end-of- fourth-grade standardized total math score served as the posttest variable, and the 
end-of-third-grade standardized total math score served as the pretest. In most districts, students in first 
grade are not given standardized tests. Because we did not have pretest data for the second grade students, 
we decided not to analyze the second-grade student data using multilevel models. 

Sample Size and Missing Data 

Table 20 provides information on the number of students who had standardized mathematics 
achievement data available. As is evident from this table, there was a fair amount of missing 
standardized-test-score data, either because it was missing district-wide, or because specific students were 
not tested. Data from some districts were not available for several reasons, including (a) some grade 
levels were not tested in some districts, (b) a computer system change in one district between years 1 and 
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2 of the project interfered with getting the data, and (c) one of the district coordinators responsible for 
providing the student assessment scores left the district too late for a replacement to be found. 

Table 20 



Number of Students with Valid Standardized Mathematics Achievement Data by Grade and District 



District 




Grade 2 


Grade 4 


Grade 6 




Pretest 


Posttest 


Pretest 


Posttest 


Pretest 


Posttest 


1 


Valid N 


- 


132 


163 


184 


93 


110 




Missing 


- 


77 


159 


138 


103 


86 


2 


Valid N 


- 


0 


0 


126 


101 


112 




Missing 


- 


97 


127 


1 


22 


11 


3 


Valid N 


- 


49 


107 


111 


29 


8 




Missing 


- 


0 


12 


8 


29 


28 


4 


Valid N 


- 


0 


0 


142 


0 


64 




Missing 


- 


146 


175 


33 


103 


39 


5 


Valid N 


- 


75 


56 


66 


79 


92 




Missing 


- 


11 


13 


3 


26 


13 



For consistency, only fourth-grade and sixth-grade students who were in the MPP analyses were 
included in the standardized-test-score analysis. In addition, for the standardized-test-score analyses, only 
students with both “pretest” and “posttest” standardized-test scores were included. 

Because of the smaller sample sizes, there were many classrooms that had no ELL students. Thus, for 
the standardized-test-score analyses, the ELL status variable was not evaluated. 

Transforming the Standardized-Test Scores to the Same Metric 

Because different standardized achievement tests were used, and because the test score data were 
reported in different metrics, the first step in the data analysis was to transform all data to standard scores, 
also known as z-scores. For each district and grade level, “pretest” scores were transformed to z-scores 
using the mean and standard deviation of the pretest for that district and grade level. Likewise, for each 
district and grade level, “posttest” scores were transformed to z-scores using the mean and standard 
deviation of the posttest for that district and grade level. Recall that z-scores have a mean and standard 
deviation of zero and one, respectively. Thus, for fourth-grade students in District 1 , for example, the 
mean and standard deviation of the pretest z-scores scores would be zero and one respectively, and the 
mean and standard deviation of the posttest z-scores scores would also be zero and one respectively. 

Individual z-scores tend to be very small values, ranging mostly between ±3 . Because it is difficult to 
look at tables of results with such small values, we transformed these z-scores to T-scores, with a mean of 
50 and a standard deviation of 10, using the formula 7] = 50 + 10(z/) . Thus, for example, for fourth- 
grade students in District 1, the mean and standard deviation of the pretest T-scores would be 50 and 10 
respectively, and the mean and standard deviation of the posttest T-scores would also be 50 and 10 
respectively. 

There are two issues to note in interpreting these standardized-test-score results. First, because the 
standardized-test scores were transformed to T-scores within each district, any treatment effect needs to 
be interpreted as the impact on student performance relative only to the performance of students in the 
district. Second, because different tests were used and different metrics reported, we cannot look at 
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“growth” over the course of a year. For any given grade level and district, the mean pretest score will be 
50, and the mean posttest score will be 50. Thus, for the standardized-test-score analyses, it is not 
meaningful to evaluate the difference between the mean pretest and mean posttest scores. 



Multilevel Analyses of the Standardized Achievement Test Results 

Two multilevel analyses of the standardized-test scores were performed, one for the fourth-grade 
students, and one for the sixth-grade students. For each of these multilevel analyses, the pretest score 
served as a level- 1 covariate, and the experimental group served as the level-2 predictor. For both the 
fourth-grade and sixth-grade multilevel analyses, to aid in the interpretation of the multilevel analysis 
results, the pretest variable was centered by subtracting the pretest mean from each student’s pretest 
score. 

Descriptive statistics for the standardized-test-score analyses are presented in Table 21. For the 
fourth-grade students, no statistically significant difference was found in the adjusted posttest means 
between the experimental and control groups {b = 1.09, z = 0.63,/) = 0.528). Likewise, for the sixth- 
grade students, no statistically significant difference was found in the adjusted means between the 
experimental and control groups {b = .102, z = 0.07,/) = 0.946). (Recall that for the second grade 
students, no multilevel analyses were performed). 

Table 21 



Descriptive Statistics for Standardized Tests by Treatment Group and Grade 



Treatment Group 


Pretest 


Posttest 


Grade 2 


Control 


Mean 




51.1 




SD 




9.1 




N 




137 


Experimental 


Mean 




48.7 




SD 




10.7 




N 




119 


Grade 4 


Control 


Mean 


48.5 


48.4 




SD 


9.9 


10.0 




N 


186 


186 


Experimental 


Mean 


52.2 


52.2 




SD 


9.8 


9.5 




N 


131 


131 


Grade 6 


Control 


Mean 


51.8 


51.3 




SD 


9.8 


9.6 




N 


158 


158 


Experimental 


Mean 


Al.l 


48.4 




SD 


9.7 


10.2 




N 


124 


124 



Note. Standardized test scores were converted to T-scores with a mean and SD 
of 50 and 10, respectively for both the pretest and posttest. For these 
standardized test score analyses, one cannot evaluate growth from pretest to 
posttest. 
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Implementation Study Results 



As reported above, analyses of students’ Pitfalls Quiz scores showed that students in the experimental 
groups at all three grade levels outperformed the control group students. Attributing these differences to 
MPP requires evidence that establishes that the lessons were implemented in experimental classrooms. In 
addition, it is important to look for systematic differences in lesson implementation in higher- versus 
lower-scoring classrooms, in order to understand the conditions that enhance students’ math learning. 
Therefore, the research also examined the fidelity of lesson implementation as enacted within MPP 
classrooms, compared to the structure and processes that were intended by the curriculum designers. 
Teacher questionnaires and audio-recorded classroom discourse were analyzed to address the following 
questions: 

• How closely does MPP as enacted follow the structure, content, and discourse processes that 
were intended by the curriculum designers? 

• How does MPP as enacted in classrooms that had greater student math score gains compare with 
lessons in classrooms with lower student gains? 

Lesson Components Implemented 

Completion of major components within the lessons. Each MPP lesson includes several components 
that are done over a period of two days. Table 22 lists these components, and shows the proportion of 
components in Lessons 5 and 6 that were reported as completed by the 1 1 teachers in the implementation 
study. This table also disaggregates the data by student scores on the MPP Pitfalls Quizzes, showing 
results separately for “higher-scoring” than “lower-scoring” classes. Classes were considered higher- 
scoring if they had a mean gain from pre- to posttest of at least 15 percentage points and a post mean 
score of at least 40% correct, in contrast to lower-scoring classrooms that did not meet those cutoffs. As 
shown in Table 22, almost all teachers in the implementation study reported completing all parts of the 
lessons on both days of each lesson, with just one notable exception. The last component of the Day 1 
lessons. Look Back, in Lesson 5, was not done in two of the lower-scoring classes and one of the higher- 
scoring classes. (Data were not available for this component of Lesson 6 because of an omission on the 
questionnaire.) The omission of this component may be attributable in part to teachers’ running out of 
time to complete the last part of the Day 1 lesson. Another possible explanation is that, compared to other 
components of the lesson, the Look Back component is not emphasized as strongly in the teachers' 
introduction to MPP. 

Audio recordings of Lesson 6 in all of these classrooms were also analyzed for presence or absence of 
lesson components, providing additional evidence in support of teachers’ self-reports (see Table 23). 
Because of technical problems, some lessons were not recorded for some teachers. However, as the 
teachers also reported on the questionnaires, almost all of the teachers were observed to complete all of 
the major parts of the lesson. Once again, the Look Back component was most frequently skipped. In 
addition, several other components were also not always completed, particularly in lower-scoring 
classrooms. 

Audio recordings were also examined in more detail to determine how completely some of the multi- 
step lesson components were implemented. These analyses also indicate that the teachers did not adhere 
rigidly to the lesson “scripts”, instead they varied them to some extent. For example, on Day 1, across all 
classes, roughly half of the prompts in Discussing the OK and Discussing the Pitfall were completed. 
Although the numbers are very small, for Lesson 6 it is worth noting that higher-scoring classrooms 
completed more parts of these two discussions, which are core activities of the MPP approach. 
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Table 22 



Percent of Lesson Components Completed in Higher- and Lower-Scoring Classrooms^ 



Lesson component 


Low (« 


= 5) 


High (k 


= 6) 


Total (/V 


= 11) 


Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


Day 1 


Purpose 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Math Words 


100.0% 


100.0% 


83.5% 


100.0% 


90.9% 


100.0% 


Starter Problem 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Discussing the OK 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Discussing the Pitfall 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Things to Remember 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Look Bade’ 


60.0% 


- 


83.5% 


- 


72.7% 


- 


Day 2 


Review of Day 1 


100.0% 


100.0% 


83.3% 


100.0% 


90.9% 


100.0% 


Our Turn 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


Your Turn 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 


100.0% 



Note. Data reported by teachers in Implementation Study Questionnaire. 

“Higher- scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of 
>40% correct. 

'’Item not included on Implementation Study Questionnaire for Lesson 6 



Table 23 

Percent of Lower- and Higher-Scoring Classroom^' in which Components of MPP Lesson 6 Were Present in 



Audio Recordings 



Lesson component 




Low 

(k = 5) 


High 

(k = 6) 


Total 

(/V=ll) 




Day 1 








Discussion Builders 




100.0% 
{n =4) 


100.0% 
(« =4) 


100.0% 

(/V=8) 


Purpose 




80.0% 


100.0% 
(« = 5) 


90.0% 
{N= 10) 


Math Words 




80.0% 


100.0% 
(« = 5) 


90.0% 
{N= 10) 


Starter Problem 




80.0% 


100.0% 
(n = 5) 


90.0% 
(N= 10) 


Discussing the OK 




100.0% 


100.0% 


100.0% 


be 

Mean percent of lesson prompts present 


Mean 


52.0% 


59.0% 


55.8% 




(SD) 


(24.3%) 


(23.0%) 


(22.7%) 


Discussing the Pitfall 




50.0% 
{n =4) 


100.0% 
(« = 5) 


77.8% 

{N=9) 


Mean percent of lesson prompts present (of 


Mean 


60.0% 


88.0% 


80.0% 


five possible)*”* 


(SD) 


(56.6%) 


(17.9%) 


(30.6%) 


Things to Remember 




100.0% 
{n =4) 


33.3% 


60.0% 
{N= 10) 


Mean percent of lesson prompts present (of 


Mean 


50.0% 


100.0% 


66.7% 


three possible)*’ 


(SD) 


(19.2%) 


(0.0%) 


(29.8%) 


Look Back 




25.0% 
(« = 4) 


66.7% 


50.0% 
{N= 10) 
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Day 2 



Review of Day 1 


100.0% 


100.0% 


100.0% 


Our Turn 


100.0% 


100.0% 


100.0% 


Percent who completed all three problems 


100.0% 


83.3% 


90.9% 


Percent who completed all three problems 


80.0% 


83.3% 


81.8% 


Your Turn 


100.0% 


100.0% 


100.0% 




{n =4) 


(« =4) 


(A =8) 


Percent who completed all three problems 


75.0% 


75.0% 


75.0% 




(« = 4) 


(« = 4) 


(A =8) 


Percent who completed all three problems 


75.0% 


66.7% 


71.4% 




(« = 4) 


(« = 3) 


(A =7) 



Note. No classroom completed all of the lesson components. 

‘‘Higher-scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of 
>40% correct. 

'’Mean percent only in classes where component ws present. 

“Numbers of components in Discussing the OK for each grade: Grade 2=17, Grade 4=18, and Grade 6 = 20. 
‘'Numbers of components in Discussing the Pitfall is five for all grades. 



The Day 1 activity of reviewing Things to Remember at three points in the lesson was partially 
implemented as well, but with lower-scoring classrooms completing more of these reviews. 

On Day 2, all lessons observed began with a review of the previous day’s conclusions, and then in the 
Our Turn and Your Turn sections, the classes worked on problems like those discussed on Day 1 . More of 
the classes completed all three Our Turn problems than the Your Turn, and, in general, only some classes 
completed all of the practice problems. 

Length of Lessons 

Questions about lesson length were included on the Implementation Study Questionnaire (see 
Table 24). MPP lessons are designed to be completed in less than two class periods - roughly one period 
on Day 1, and a shorter time on Day 2. Teachers’ reports indicate that lesson length approximates these 
intentions. Teachers estimated that the Core Lesson (including both Day 1 and Day 2) for Lesson 5 
averaged close to one hour (57.3 minutes), and for Lesson 6, approximately 20 additional minutes (81.6 
minutes). Time required for Lesson 5 ranged from less than one class period to two periods; time for 
Lesson 6 ranged from one class period to more than two. 

Across both the Core Lessons and each of the Mini Lessons, there is a consistent trend of lessons 
taking longer in the lower-scoring classes than the higher-scoring. 

Discourse Processes 

Audio recordings of lessons and teachers’ questionnaire responses provide a variety of information 
about the interactions that comprise MPP. Table 25 shows teacher and student discourse behaviors that 
were observed in the classroom recordings. Although the number of classrooms is small and it is not 
possible to reach conclusions about differences with any certainly, we point out here some of the 
strongest patterns that were observed. 
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Table 24 



Length of MPP Lessons in Lower- and Higher-Scoring Classrooms’^ 



Questionnaire item 


Low 
(k = 5) 


High 
(« = 6) 


Total 
(A= 11) 




Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


1.3. About how long did you take to teach 
the core lesson, which does not 
include the mini-lessons? Note: A 
class period is about 45-50 minutes. 


a. Less than 1 class period 


20.0% 


0.0% 


16.7% 


0.0% 


18.2% 


0.0% 


b. 1 class period 


20.0% 


60.0% 


50.0% 


33.3% 


36.4% 


45.5% 


c. > 1 < 2 class periods 


60.0% 


20.0% 


33.3% 


66.7% 


45.5% 


45.5% 


d. > 2 class periods 


0.0% 


20.0% 


0.0% 


0.0% 


0.0% 


9.1% 


2.3a. Core Lesson - Approximate minutes 


Mean 


60.0 


89.0 


55.0 


75.5 


57.3 


81.6 


SD 


19.0 


34.0 


27.6 


30.3 


23.1 


31.2 


Median 


70.0 


90.0 


42.5 


72.5 


45.0 


75.0 


Range 


35-80 


50-125 


30-90 


43-120 


30-90 


43-125 


2.3b. Mini Lesson 1 - Approximate minutes 


Mean 


25.0 


28.8 


20.0 


16.7 


22.0 


21.5 


SD 


15.8 


13.1 


7.1 


2.6 


10.9 


10.0 


Median 


22.5 


32.5 


20.0 


15.0 


20.0 


17.5 


Range 


10-45 


10-40 


10-30 


15-20 


10-45 


10-40 


2.3c. Mini Lesson 2 - Approximate minutes 


Mean 


26.3 


22.5 


19.2 


16.7 


22.0 


19.0 


SD 


14.9 


11.9 


7.4 


2.6 


10.9 


7.7 


Median 


25.0 


22.5 


17.5 


15.0 


20.0 


15.0 


Range 


10-45 


10-35 


10-30 


15-20 


10-45 


10-35 



Note. Data reported by teachers in Implementation Study Questionnaire. 

‘‘Higher-scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of 
>40% correct. 



Number of follow-up prompts. To achieve in-depth student discussions and explanations, it is 
essential that the teacher follow up on student responses with prompts that build on or require elaboration 
of the students’ original responses. As shown in Table 25, the mean number of follow-up prompts was 
considerably greater during the Discussing the OK component (mean of 43.6 follow-up prompts per 
lesson) than in the Discussing the Pitfall (mean of 16.8 per lesson). Furthermore, the number of follow-up 
prompts during Discussing the OK in lower-scoring classrooms was greater than in higher-scoring 
classrooms (52.0 vs. 36.7). Although the reverse was true for Discussing the Pitfall (12.3 in lower-scoring 
classrooms vs. 20.4 in higher), these means are closer and less notable. This finding might be expected, 
however, since all lessons have roughly twice as many prompts for Discussing the OK component as for 
Discussing the Pitfall. Thus there are fewer prompts to follow up on. 

Number of follow-up prompts. To achieve in-depth student discussions and explanations, it is 
essential that the teacher follow up on student responses with prompts that build on or require elaboration 
of the students’ original responses. As shown in Table 25, the mean number of follow-up prompts was 
considerably greater during the Discussing the OK component (mean of 43.6 follow-up prompts per 
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lesson) than in the Discussing the Pitfall (mean of 16.8 per lesson). Furthermore, the number of follow-up 
prompts during Discussing the OK in lower-scoring classrooms was greater than in higher-scoring 
classrooms (52.0 vs. 36.7). Although the reverse was true for Discussing the Pitfall (12.3 in lower-scoring 
classrooms vs. 20.4 in higher), these means are closer and less notable. This finding might be expected, 
however, since all lessons have roughly twice as many prompts for Discussing the OK component as for 
Discussing the Pitfall. Thus there are fewer prompts to follow up on. 

Table 25 



Mean Percent and Frequency of Lower- and Higher-Scoring Classrooms^' in which Components of MPP 
Lesson Were Present in Audio Recordings 



Lesson component 


Low 


High 


Total 


Mean 


iSD) 


Mean 


iSD) 


Mean 


iSD) 


Day 1 


Discussing the OK 


(« 


= 5) 


in 


= 6) 


iN 


= 11) 


Number of follow-up prompts 


52.0 


(24.6) 


36.7 


(17.5) 


43.6 


(21.4) 


Number of Discussion Builders used by teacher 


4.0 


(3.5) 


2.7 


(2.9) 


3.3 


(3.1) 


Number of Discussion Builders used by students 


1.4 


(1.7) 


1.8 


(1.5) 


1.6 


(1.5) 


Times students were asked to talk with neighbor 


2.4 


(0.9) 


2.7 


(3.6) 


2.5 


(2.6) 


Times students were asked to show/explain their ideas 


6.8 


(4.3) 


4.0 


(4.4) 


5.3 


(4.4) 


Rating of length of student responses in discussions'* 
Discussing the Pitfall 


2.4 


(0.8) 


3.2 


(1.0) 


2.8 


(0.9) 


Number of follow-up prompts 


12.3 


(17.0) 


20.4 


(20.0) 


16.8 


(18.1) 


Number of Discussion Builders used by teacher 


0.0 


(0.0) 


2.2 


(2.3) 


1.2 


(2.0) 


Number of Discussion Builders used by students 


0.0 


(0.0) 


1.2 


(1.6) 


0.7 


(1.3) 


Times students were asked to talk with neighbor 


1.8 


(2.9) 


1.0 


(1.2) 


1.3 


(2.0) 


Times students were asked to show/explain their ideas 


0.3 


(0.5) 


1.0 


(1.4) 


0.7 


(1.1) 


Rating of length of student responses in discussions'* 


2.0 


(1.8) 


4.5 


(0.7) 


3.4 


(1.8) 


Day 2 


Our Turn 


Number of Discussion Builders used by teacher 


8.6 


(7.8) 


8.5 


(5.4) 


8.5 


(6.2) 


Number of Discussion Builders used by students 


2.8 


(2.8) 


2.5 


(3.0) 


2.6 


(2.8) 


Times students were asked to talk with neighbor 


5.0 


(2.6) 


8.5 


(9.6) 


6.9 


(7.2) 


Times students were asked to show/explain their ideas 


7.8 


(4.6) 


5.0 


(3.0) 


6.3 


(3.9) 


Rating of length of student responses in discussions'* 


1.9 


(1.2) 


2.6 


(0.9) 


2.3 


(1.1) 


Your Turn 


(« 


= 3) 


(« 


= 4) 


iN 


= 7) 


Number of Discussion Builders used by teacher 


1.7 


(2.1) 


4.8 


(6.4) 


3.4 


(5.0) 


Number of Discussion Builders used by students 


2.0 


(3.5) 


3.8 


(5.6) 


3.0 


(4.5) 


Times students were asked to talk with neighbor 


0.0 




0.0 




0.0 




Times students were asked to show/explain their ideas 


0.0 




0.8 


(1.0) 


0.4 


(0.8) 


Rating of length of student responses in discussions'* 


1.0 


(0.8) 


1.6 


(1.3) 


1.3 


(1.0) 



‘‘Higher-scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of 
>40% correct. 

'’Length of student responses rated on a scale from 1-Few if any “moderate-to-long” student responses, to 5-Many 
or mostly “moderate-to-long” student responses. 
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Number of Discussion Builders. MPP incorporates Discussion Builders to support teachers and 
students in achieving extended classroom discourse about the math in each lesson. As shown in Table 25, 
the number of times that teachers used Discussion Builders averaged just over 3 during Discussing the 
OK, and fewer than 2 during Discussing the Pitfall. In contrast, the teacher used Discussion Builders 
more than 8.5 times per lesson during the Our Turn segments. This pattern also holds for the number of 
times students used Discussion Builders in these three parts of the lessons. During the Your Turn 
segments. Discussion Builders were used approximately as frequently as during Discussing the OK. There 
are few notable differences between lower- and higher- scoring classes in Discussion Builder use, except 
during Your Turn when discussions in higher-scoring classes included more of these sentence types than 
the lower-scoring classes. This finding is encouraging, since it indicates that the Discussion Builders are 
spontaneously used, even for lesson segments that are not guided by specific prompts in the teaching 
guide (such as the Our Turn). Since one of the goals of MPP is for students to become more proactive in 
their learning, this finding could be an indication that the Discussion Builders may play a role in that 
desired behavior. 

Teachers were also asked about their use of Discussion Builders on the Implementation 
Questionnaire. As shown in Table 26, the majority of teachers reported that their students use the 
Discussion Builders one to five times during a typical day. More teachers of lower-scoring students than 
higher-scoring perceived their students as using Discussion Builders more than five times a day (40% vs. 
16.7%). More than 80% of the teachers reported that their students use these sentence forms both in math 
and other subject areas. 

Table 26 



Percent of Teachers Reporting How Students Use MPP Discussion Builders in Lower- and Higher-Scoring 
Classrooms^ 



Questionnaire item 


Low 

(k = 5) 


High 

(k = 6) 


Total 

(A=ll) 


1 . When do your students use the Discussion Builders? 


a. The children use the Discussion Builders in math and other subject 


80.0% 


83.3% 


81.8% 


areas. 








b. The children use the Discussion Builders only in math. 


0.0% 


16.7% 


9.1% 


c. The children seldom use the Discussion Builders in any subject area. 


20.0% 


0.0% 


9.1% 


2. Estimate about how often you think the students use the Discussion 








Builders during a typical day. 








a. More than 1 5 times a day 


0.0% 


0.0% 


0.0% 


b. Between 5 and 15 times a day 


40.0% 


16.7% 


27.3% 


c. 1 to 5 times a day 


60.0% 


66.7% 


63.6% 


d. Never 


0.0% 


16.7% 


9.1% 



Note. Data reported by teachers in Implementation Study Questionnaire. 

“Higher- scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of 
>40% correct. 



Times students were asked to talk with their neighbor. Periodically during the lessons, teachers are 
urged to have students talk about the math with one another. Analysis of the audio recordings indicate that 
this occurred approximately 2-3 times during Discussing the OK, and 1-2 times during Discussing the 
Pitfall, but most frequently by far (close to 7 times per lesson, on average) during Our Turn on Day 2. As 
would be expected, students never were asked to talk among themselves during Your Turn. Students 
talked among themselves more often in higher-scoring classrooms during Our Turn (8.5 vs. 5.0 times per 
lesson, on average), but otherwise lower- and higher-scoring classes were very similar. 
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Times students were asked to show or explain their math ideas. A key aspect of the MPP approach 
is fostering students’ abilities to articulate their math thinking. Teachers explicitly asked students to 
explain their ideas primarily during Discussing the OK (mean of over 5 times per lesson) and Our Turn 
(mean of over 6 times per lesson). During both of these segments, students in lower-scoring classes were 
asked to explain their thinking more frequently than those in higher-scoring classes. 

Length of student responses in discussions. The length of students’ statements in each audio 
recording was rated on a scale from 1 - “Few if any ‘moderate-to-long’ student responses,” to 5 - “Many 
or mostly ‘moderate-to-long’ student responses.” These ratings were assigned within each lesson 
component. As shown in Table 25, during every lesson component, students in higher-scoring classes 
gave longer responses than those in lower-scoring classes. Overall, students in lower-scoring classes were 
rated approximately 2 on average, whereas those in higher-scoring classes were rated 3 or higher. 

Teachers’ Reports on Lesson Implementation 

Teachers reported the following on questionnaires (see Tables 27 and 28). 

Use of Getting Started tasks. As shown in Table 28, teachers at all three grade levels agreed on 
average with a statement that they used the MPP Getting Started tasks at the beginning of the year to help 
their students learn how to use the Discussion Builders. Second grade teachers agreed most strongly with 
this statement. 

Use of Mini-Lessons. Teachers at all three grade levels agreed strongly with statements that they used 
all of the Mini-Lesson 1 and 2 lessons, involving multiple-choice questions and explanations in writing 
from their students (see Table 28). 

Use of the Mathematical Background section. As shown in Tables 27 and 28, the majority of 
teachers reported reading closely the Mathematical Background section of the teaching guide for Lessons 
5 and 6, and reading this section for almost every lesson. Most teachers of higher-scoring classes reported 
reading this section closely for both Lesson 5 and 6 (with the highest percentage for Lesson 6), whereas 
most teachers of lower-scoring classes Just skimmed the Mathematical Background section, or did not 
read it, particularly for Lesson 6. 

Questions in the teaching guide. Questionnaire responses summarized in Tables 27 and 28 indicate 
clearly that teachers use both the questions in the teaching guide and some of their own when conducting 
discussions during MPP lessons. 

Teacher behavior during students’ paired discussions. Almost all of the teachers reported that when 
their students were talking among themselves about the math, the teachers walked around and sometimes 
interacted with students about their work. 
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Table 27 

Teachers’ Use of MPP in Lower- and Higher-Scoring Classrooms^ 





Low (h 


= 5) 


High (« 


= 6) 


Total {N 


= 11) 




Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


Lesson 5 


Lesson 6 


3. How did you use the Mathematical Background section of the teaching 
guide for Lesson 5/6? 


a. I read it closely prior to teaching the lesson. 


60.0% 


20.0% 


66.7% 


83.3% 


63.6% 


54.5% 


b. I skimmed it prior to teaching the lesson. 


40.0% 


60.0% 


33.3% 


16.7% 


36.4% 


36.4% 


c. I did not read it. 


0.0% 


20.0% 


0.0% 


0.0% 


0.0% 


9.1% 


d. Other 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


4. How did you use the questions in the teaching guide for Lesson 5/6? 


a. I used all or most of the questions in the teaching guide and none of my 
own. 


0.0% 


0.0% 


16.7% 


16.7% 


9.1% 


9.1% 


b. I used all or most of the questions in the teaching guide and some of my 
own. 


100.0% 


80.0% 


83.3% 


66.7% 


90.9% 


72.7% 


c. I used few or none of the questions in the teaching guide and mostly 
used ones I made up. 


0.0% 


20.0% 


0.0% 


16.7% 


0.0% 


18.2% 


d. Other 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


5. What did you do while your students were in paired discussions? 


a. Walked around and sometimes interacted with students about their 
work. 


100.0% 


80.0% 


100.0% 


100.0% 


100.0% 


90.9% 


b. Walked around but seldom interacted with students about their work. 


0.0% 


20.0% 


0.0% 


0.0% 


0.0% 


9.1% 


c. I did not walk around or interact with students about their work but 
waited for them to finish before proceeding. 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


d. The students did not have paired discussions. 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 


0.0% 



Note. Data reported by teachers in Implementation Study Questionnaire. 

“Higher-scoring classroom = a mean gain from pre- to posttest of >15 percentage points and a post mean score of >40% correct, 
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Table 28 

Teachers’ Self-Reported Use of MPP in 2003-04 End-of-Year Questionnaire 



End-of-year questionnaire item 




Grade 2 
(n = 18) 


Grade 4 
(n = 25) 


Grade 6 
(« = 13) 


Total 

(W=56) 


Teacher Use of MPP Materials 


14. I used most of the questions in the teaching guides, along 
with some of my own, to help me conduct the discussions 
with my class. 


Mean 

(SD) 

Range 


8.5 

(1.7) 

4-10 


8.5 

(1.6) 

4-10 


8.8 

(1.4) 

6-10 


8.6 

(1.6) 

4-10 


15. I made up most of my own questions to conduct the 
discussion with my class, instead of using those in the 
teaching guide. 


Mean 

(SD) 

Range 


3.1 

(2.5) 

1-10 


4.4 

(2.7) 

1-10 


3.1 

(2.6) 

0-8 


3.7 

(2.7) 

0-10 


18. I used the Getting Started tasks at the beginning of the 
school year to help my students learn how to use the 
Discussion Builders. 


Mean 

(SD) 

Range 


8.2 

(2.4) 

1-10 


6.6 

(3.7) 

0-10 


7.3 

(2.9) 

0-10 


7.3 

(3.2) 

0-10 


22. I used all of the Mini-Lesson 1 lessons (multiple-choice 
questions) with my students. (Write a 10 if you don't think 
you skipped any.) 


Mean 

(SD) 

Range 


9.1 

(1.6) 

5-10 


9.4 

(1.3) 

4-10 


9.5 

(1.0) 

7-10 


9.4 

(1.3) 

4-10 


23. I used all of the Mini-Lesson 2 lessons (requiring 

explanations in writing) with my students. (Write a 10 if 
you don't think you skipped any.) 


Mean 

(SD) 

Range 


8.9 

(1.8) 

4-10 


9.2 

(1.7) 

4-10 


9.4 

(1.2) 

6-10 


9.1 

(1.6) 

4-10 


25. I read the Mathematical Background section of the 

teaching guide for almost every lesson. (Write a 10 if you 
don't think you skipped any.) 


Mean 

(SD) 

Range 


8.8 

(1.7) 

4-10 


8.8 

(2.0) 

4-10 


8.8 

(2.5) 

1-10 


8.8 

(2.0) 

1-10 



Note. Ratings on 10-point scale from 1 -Don’t agree at all, to 10-Strongly agree. 



Teachers ’ Evaluation of MPP 

Teachers were asked to rate the impact and value of MPP on the end-of-year questionnaire. Results 
are provided in Table 29. 

Impact on student learning. For the most part, teachers agreed rather strongly with a variety of 
statements that MPP was helpful for their students (see Table 29). These statements included opinions 
that, after completing the set of lessons this year, their students understood the math topics in the lessons 
better than students in past years, all of their students are making fewer pitfalls, and MPP helped most of 
their students learn the math concepts and prevent pitfalls. The one statement that teachers at all grade 
levels disagreed with is that the lessons were helpful for their non-native English-speaking students. 
These opinions are contradicted by the fact that the lessons actually were at least as effective for ELLs’ 
math achievement as for English-proficient students. Teachers’ ratings with respect to special needs 
students were stronger than for ELL students. 

Interestingly, sixth-grade teachers agreed but less strongly that the lessons were helpful for high- 
performing students, whereas fourth-grade teachers agreed less strongly that they were helpful for low- 
performing students. Second-grade teachers agreed but less strongly than the other grades with a 
statement that their students understood the math topics in the lessons better than students in past years. 

Teachers’ opinions of students’ opinions. Teachers at all grade levels agreed strongly that their 
students really liked Math Pathways and Pitfalls and that the format of the lessons was easy for most of 
their students to follow. Second and fourth grade teachers agreed that their students really liked the video 
of other students doing a Math Pathways and Pitfalls lesson, but sixth-grade teachers disagreed with this 
statement. 
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Table 29 



Teachers’ Ratings of MPP in End-of-Year Questionnaire 





Questionnaire Item 




Grade 2 
(n = 18) 


Grade 4 
(n = 25) 


Grade 6 
in = 13) 


Total 

(W=56) 


Impact on Student Learning 


4. 


These lessons were helpful for my high-performing 
students. 


Mean 

(SD) 

Range 


7.0 

(2.4) 

2-10 


8.2 

(1.5) 

5-10 


5.9 

(2.5) 

2-9 


7.3 

(2.2) 

2-10 


5. 


These lessons were helpful for my low-performing 
students. 


Mean 

(SD) 

Range 


6.7 

(2.6) 

3-10 


5.8 

(2.8) 

1-10 


7.9 

(1.7) 

4-10 


6.6 

(2.6) 

1-10 


6. 


These lessons were helpful for my non-native English- 
speaking students. 


Mean 

(SD) 

Range 


4.5 

(3.6) 

0-9 


3.4 

(3.3) 

0-9 


3.5 

(4.1) 

0-10 


3.8 

(3.6) 

0-10 


7. 


These lessons were helpful for my special needs students. 


Mean 

(SD) 

Range 


5.2 

(3.1) 

0-9 


5.6 

(2.8) 

0-9 


6.2 

(3.7) 

0-10 


5.6 

(3.1) 

0-10 


8. 


When working on the Starter Problems, many of my 
students made the same or similar pitfalls to the ones in 
the lessons. 


Mean 

(SD) 

Range 


7.1 

(2.2) 

4-10 


8.3 

(1.8) 

3-10 


6.7 

(2.8) 

2-10 


7.5 

(2.2) 

2-10 


9. 


After completing the set of lessons this year, my students 
understood the math topic in the lessons better than 
students in past years. 


Mean 

(SD) 

Range 


5.8 

(2.2) 

0-10 


7.0 

(2.4) 

0-10 


6.8 

(2.0) 

3-10 


6.6 

(2.3) 

0-10 


10. 


After completing the set of lessons this year, all of my 
students are making fewer pitfalls. 


Mean 

(SD) 

Range 


6.1 

(1.3) 

4-8 


7.2 

(1.5) 

4-10 


6.6 

(2.3) 

2-10 


6.7 

(1.7) 

2-10 


12. 


The Discussion Builders were very important in helping 
my students learn how to discuss mathematics. 


Mean 

(SD) 

Range 


7.4 

(2.5) 

3-10 


7.2 

(2.5) 

1-10 


9.2 

(1.3) 

6-10 


7.8 

(2.4) 

1-10 


13. 


The Discussion Builders helped my students learn to 
discuss other content areas besides math. 


Mean 

(SD) 

Range 


7.2 

(2.5) 

2-10 


7.5 

(2.4) 

1-10 


8.5 

(2.8) 

0-10 


7.6 

(2.5) 

0-10 


20. 


The video played an important role in helping my students 
learn how to do the Math Pathways and Pitfalls lessons. 


Mean 

(SD) 

Range 


7.3 

(2.8) 

0-10 


7.0 

(2.9) 

0-10 


4.6 

(4.0) 

0-10 


6.5 

(3.3) 

0-10 


24. 


The Mini-Lessons that I used helped my students 
consolidate their understanding. 


Mean 

(SD) 

Range 


7.6 

(2.0) 

3-10 


7.9 

(1.2) 

5-10 


8.7 

(1.5) 

5-10 


8.0 

(1.6) 

3-10 


30. 


I believe that the Math Pathways and Pitfalls lessons 
helped most of my students learn the math concepts and 
prevent pitfalls. 


Mean 

(SD) 

Range 


7.0 

(1.9) 

4-10 


7.9 

(1.8) 

3-10 


7.3 

(2.0) 

2-10 


7.5 

(1.9) 

2-10 


Students’ Opinions 


1. 


Most of my students really liked the Math Pathways and 
Pitfalls lessons. 


Mean 

(SD) 

Range 


7.3 

(1.5) 

4-10 


7.5 

(2.2) 

0-10 


6.9 

(2.0) 

4-9 


7.3 

(1.9) 

0-10 


2. 


The format of the lessons was easy for most of my 
students to follow. 


Mean 

(SD) 

Range 


7.7 

(1.9) 

3-10 


7.1 

(2.3) 

1-10 


8.6 

(1.7) 

5-10 


7.7 

(2.1) 

1-10 


19. 


My students really liked the video of other students doing 
a Math Pathways and Pitfalls lesson. (Write NA if you 
didn't show the video.) 


Mean 

(SD) 

Range 


7.5 

(2.6) 

0-10 


6.9 

(3.3) 

0-10 


4.2 

(3.9) 

0-10 


6.5 

(3.4) 

0-10 
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Questionnaire Item 




Grade 2 
(n = 18) 


Grade 4 
{n = 25) 


Grade 6 
(« = 13) 


Total 

(A^=56) 


Teachers’ Opinions of MPP 


3. The teaching guide is clearly organized and easy to 
follow. 


Mean 

(SD) 

Range 


l.A 

(2.1) 

3-10 


7.0 

(2.7) 

2-10 


8.8 

(1.5) 

5-10 


7.6 

(2.4) 

2-10 


1 1 . The language support provided in the teaching guide and 
in the student lessons (for example, the Math Words) was 
helpful. 


Mean 

(SD) 

Range 


8.3 

(1.6) 

5-10 


7.7 

(1.6) 

2-10 


7.6 

(2.2) 

3-10 


7.9 

(1.8) 

2-10 


16. Prior to the Math Pathways and Pitfalls lessons, 1 

conducted thought-provoking math discussions with my 
class several times a week. 


Mean 

(SD) 

Range 


5.1 

(2.8) 

1-9 


5.6 

(2.4) 

1-10 


6.6 

(2.7) 

2-10 


5.7 

(2.6) 

1-10 


17. 1 learned a lot about using thought-provoking discussions 
to teach math through the Math Pathways and Pitfalls 
lessons. 


Mean 

(SD) 

Range 


8.1 

(1.6) 

5-10 


7.5 

(1.4) 

4-10 


7.9 

(1.5) 

6-10 


7.8 

(1.5) 

4-10 


2 1 . The language support for students provided in the teaching 
guides and in the student lessons helped me prepare 
students for the special mathematical vocabulary and 
symbols in the lesson. 


Mean 

(SD) 

Range 


8.2 

(1.8) 

5-10 


7.3 

(1.5) 

4-10 


6.7 

(2.4) 

2-10 


7.4 

(1.9) 

2-10 


26. Substituting a Math Pathways and Pitfalls lesson for two 
of my regular lessons about once a month worked well. 


Mean 

(SD) 

Range 


5.5 

(3.1) 

1-10 


6.1 

(3.1) 

0-10 


5.4 

(3.0) 

1-10 


5.8 

(3.0) 

0-10 


27. 1 would rather teach the whole set of Math Pathways and 
Pitfalls lessons as a single unit, rather than throughout the 
year. 


Mean 

(SD) 

Range 


6.2 

(3.6) 

1-10 


6.7 

(3.1) 

1-10 


6.1 

(3.8) 

1-10 


6.4 

(3.4) 

1-10 


28. 1 would love to use the Math Pathways and Pitfalls 
lessons again next year. 


Mean 

(SD) 

Range 


7.9 

(2.1) 

3-10 


8.3 

(1.7) 

5-10 


7.3 

(2.4) 

3-10 


7.9 

(2.0) 

3-10 


29. The students in our school would benefit greatly if all of 
the teachers used the Math Pathways and Pitfalls lessons. 


Mean 

(SD) 

Range 


7.2 

(1.6) 

4-10 


7.4 

(2.2) 

3-10 


7.2 

(2.0) 

2-10 


7.3 

(2.0) 

2-10 


3 1 . The Math Pathways and Pitfalls lessons supported hard- 
to-teach topics in our math textbook. 


Mean 

(SD) 

Range 


6.7 

(2.1) 

1-10 


7.5 

(2.3) 

0-10 


7.8 

(2.2) 

2-10 


7.3 

(2.2) 

0-10 


32. The Math Pathways and Pitfalls lessons went beyond our 
textbook in a positive way. 


Mean 

(SD) 

Range 


7.6 

(2.0) 

5-10 


7.9 

(2.8) 

0-10 


7.8 

(2.2) 

2-10 


7.8 

(2.4) 

0-10 


33. 1 feel that my teaching has improved as a result of using 
the Math Pathways and Pitfalls materials. 


Mean 

(SD) 

Range 


7.5 

(2.2) 

3-10 


8.0 

(1.6) 

5-10 


7.0 

(2.4) 

2-10 


7.6 

(2.0) 

2-10 



Note. Ratings on 10-point scale from 1 -Don’t agree at all, to 10-Strongly agree. 



Teachers ’ overall opinions of MPP. The teachers also agreed strongly with strong statements about 
the overall value of MPP. These include; 

• I would love to use Math Pathways and Pitfalls again next year. 

• The students in our school would benefit greatly if all of the teachers used Math Pathways and 
Pitfalls. 

• Math Pathways and Pitfalls supported hard-to-teach topics in our math textbook. 

• Math Pathways and Pitfalls went beyond our textbook in a positive way. 
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• I feel that my teaching has improved as a result of using the Math Pathways and Pitfalls 
materials. 

• 1 learned a lot about using thought-provoking discussions to teach math through Math Pathways 
and Pitfalls. 

Teachers at all grade levels, but especially at sixth grade, agreed strongly that the teaching guide is 
clearly organized and easy to follow. 

The only statement about which the teachers approached neutrality is that substituting a Math 
Pathways and Pitfalls lesson for two of their regular lessons about once a month worked well. The 
teachers agreed, instead, that they would rather teach the whole set of Math Pathways and Pitfalls lessons 
as a single unit, rather than throughout the year. 

Summary 

With respect to fidelity of lesson implementation, analysis of classroom audio recordings and teacher 
questionnaires revealed that (a) almost all teachers implemented every major component and intended 
discourse process of the lessons; (b) teachers made some minor modifications to the lesson structures — 
namely some steps or prompts were left out more than others, particularly in lower-scoring classes; 

(c) some of the tools for building extended student talk about math, such as the Discussion Builders, are 
spontaneously used by teachers and students, even for lesson segments that are not guided by specific 
prompts in the teaching guide, and during class time on subjects other than math; (d) in classes with 
higher-scoring students, there was more use of Discussion Builders by both teachers and students, were 
asked to explain their thinking less frequently than in lower-scoring classes but more often talked about 
the math among themselves, and gave longer responses about the math. 

Teachers expressed strongly positive opinions about the value of the program, including that their 
students understood the math topics in the lessons better than students in past years, that MPP helped 
most of their students learn the math concepts and prevent pitfalls, and their students really liked MPP. 
Overall, the teachers strongly agreed that they would love to use MPP again next year, and students in 
their schools would benefit greatly if all of the teachers used Math Pathways and Pitfalls. 

Conclusions 

Using a project-developed Pitfalls Quiz as the measure of mathematics achievement, this experiment 
found that student math performance in MPP classes was higher than in non-MPP classes for all three 
grade levels. For second and fourth grades, MPP benefited ELL and non-ELL students equally. The 
effect-size statistics {ESS) for second and fourth grades were .43 and .66, respectively. For sixth grade, 
MPP had a greater treatment effect for ELL students {ESS = .74) than non-ELL students {ESS = .28). In 
evaluating how equitable the impact of MPP was on students’ mathematics knowledge across levels of 
entering math knowledge, the study found no difference in the effectiveness of MPP for mathematically 
stronger versus weaker students except at fourth grade, where MPP was more effective for children who 
had higher pretest scores than for children who had lower pretest scores. Flowever, the grade 4 quiz was 
very difficult and, accordingly, less reliable than the second- and sixth-grade tests. 

For the district-administered standardized achievement tests, no statistically significant differences 
were found in the adjusted standardized achievement test posttest means between the experimental and 
control groups for either the fourth-grade students or the sixth-grade students. (No standardized 
achievement test data were available for second grade students.) 

The findings of positive impact of MPP on student mathematics performance across grades, levels of 
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English proficiency, and entering mathematics ability are consistent with an earlier study of MPP 
materials by Heller, Gordon, Paulukonis, and Kaskowitz (2000). Because the current study was based on 
a more rigorous research design (i.e., a cluster randomized design) than the one used in the Heller et. al. 
study, the results of the current study can be viewed as even stronger evidence of the effectiveness of the 
MPP materials. 

Comparing MPP Pitfalls Quiz and standardized achievement test results. Although statistically 
significant results were found for all three grade levels on the MPP Pitfalls Quizzes, no statistically 
significant results were found using standardized achievement tests as the outcome variable. This 
disparity might be due to the fact that the MPP Pitfalls Quizzes were designed to assess the rational 
number topics covered by the MPP lessons, whereas the standardized achievement tests assess a more 
global construct of mathematical achievement, so may not have been instructionally sensitive enough to 
detect differences between the MPP and non-MPP groups. 

Implementing cluster randomized designs in education. Cluster randomized designs are a powerful 
way of evaluating the impact of a given educational intervention on student learning. The random 
assignment of teachers to MPP and non-MPP groups is an important element in the internal validity of 
this study. 

There are many logistical challenges to implementing a cluster randomized design in education. First, 
random assignment of teachers requires uniformity of schedule, district policy, and preferences across 
many difference school and district contexts. Because the real world of education is so complex, there 
were many challenges involved in implementing and maintaining the research design. For example, group 
assignment dictated the timing of professional development sessions for teachers in a given group, and 
teachers’ schedules were often in conflict with the project’s. Teachers and site coordinators are highly 
mobile, resulting in considerable attrition. Furthermore, this study was carried out in several states, and 
these states differed in terms of the standardized achievement tests they used. Because school district 
officials are reluctant to add any additional standardized testing requirements over and above the tests 
they currently use, we had to rely on the standardized achievement test data provided by each district. 

In addition, because the study was conducted in multiple districts, a great deal of effort was required 
to get formal consent from each district to conduct the study. Because this study was conducted in school 
districts that were distant from each other, the project depended upon local school and district personnel 
to implement the research design. The study was vulnerable to the ongoing availability of these 
coordinators — when they moved on, communication with teachers in the district became highly 
problematic. In addition, a significant amount of time was devoted to coordinating logistical issues with 
school personnel representing the various school sites. 

Limitations. Although the MPP materials were found to have a positive impact on student learning as 
measured by the MPP Pitfalls Quiz, several limitations of the study should be noted. First, because of the 
large number of teachers who dropped out in the second year of the study, the data from the second year 
of the study were not considered usable, and we were therefore unable to compare the impact on student 
learning of one day versus four days of teacher professional development in MPP. Second, the 
standardized achievement test data were problematic. Different school districts provided different 
standardized tests to the researchers, and these different tests do not all measure the same underlying 
constructs. Moreover, there was a fair amount of missing standardized-test data, making the results of the 
analyses of these tests difficult to interpret. Finally, in terms of generalizability of the findings to other 
students, this study was implemented in five school districts across the country. Although every effort 
was made to select districts with diverse student bodies, caution is still needed in generalizing these 
results to other students. In addition, teachers participating in the study were volunteers and may not 
represent the full spectrum of teachers. 
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student name 



- or- Student ID # 



Teacher 



Date 



PftfelIsQuiz 

Fall 2003 

Rll in the circle next to the anstA/eryou choose. 

1. Which point shows the number 23? 

W X Y Z 




Grade 2 Quiz 



page 1 



MathematicsCase Methods Project/WestEd ©2003 








Find the difference 
16 

© 3 

© 7 

© 13 

© 25 



798+10 = 

© 708 

® 898 

© 808 
© 7918 



6. What number is 10 more than 482? 

© 483 

© 4812 

© 582 

© 492 
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10. 5 + 



12-4 



© 3 

© 7 

© 8 
© 11 



11. 36 

+ -*7 

© 73 

© 83 

© 713 

© 63 



12. Find the difference. 
35 - 29 = 

© 14 

© 6 

© 64 

© 7 
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VJtHte the AnsA/er 




14. Draw a point on the number line to show the number35. 








17. Ann'saunt is 23 yearsold. Her uncle is 31. What is the difference 
in their ages? 



Answer: 



18. J orge has 15 turtles and 27 fish. How many animalsdoeshe 
have in all? Show your work. 



Answer: 
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student name -or- Student ID# 

Teacher Date 

PidallsQuiz- Ractions 

Fall 2003 

Rll in the circle next to the anstA/eryou choose. 






1 

4. Which of these pictures shows 

4 


shaded? 


(D 0 






A R 




C 


© only picture A 


© 


only picture C 


® only pictures 


© 


1 

both A a nd B show 

4 
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5 




14. Which fraction IS mo re than 


? 








8 ' 




© 


5 


© 


7 






9 


15 


© 


5 


© 


2 


6 


5 
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3 




1 


17. Patty bought — 


yard of gold chain and — yard of silver chain. 




8 




4 


What is the total length of her chains? 


© 


5 


© 


1 


yards 

8 


yards 

3 


© 


— yards 


© 


— yards 


32 




12 
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student name -or- Student ID# 

Teacher Date 

PidallsQuiz- Peicerrts 

Fall 2003 

Circle the letter nextto the anstA/eryou choose. 

1. What percent of the rectangle isshaded? 

p==] 



© 


6% 


© 


60% 


© 


46% 


© 


64% 



2. What percent of the rectangle isshaded? 




© 10% 


© 150% 


© 15% 


© 1.5% 
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3. What percent of thisdiagram isshaded? 



© 25% 


© 75% 


© 34% 


© 60% 



4. What percent of the picture isshaded? 




© 


14% 


© 


33% 


© 


25% 


© 


39% 
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5. What percent of the picture isshaded? 
















© 


35% 








© 65% 


© 


60% 








© 70% 



6. Estimate the percent of students who don't have pets. 




© 


80% 


© 


30% 


© 


60% 


© 


90% 
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8. What percent is equal to —? 

8 



© 


16 % 


© 


58 % 


© 


85 % 


© 


62 . 5 % 
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9. What percent isequa I to — ? 

10 



9% 


© 


90% 


0.9% 


© 


910% 



10. What percent isequal to — ? 

4 



84% 


© 


2% 


50% 


© 


200% 
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12. Which set of numbers is in orderfrom least to g 


neatest? 


© 




1 




© 


1 




0.5 


00 


28% 


28% 

8 


0.5 


© 


1 






1 






0.5 


28% 


® 


28% — 


0.5 




00 








00 





13. Which of the following numbers is closest to 1? 




0.12 


97% 


1.2 


5 

6 


© 


0.12 




© 


1.2 


© 


97% 




© 


5 

6 



14. What is 5% of 120? 






© 


6 


© 


60 


© 


24 


© 


115 
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15. My brother has 150 cd's. 60%of them are rap music. How 
many the cd's a re rap? 



© 100 


© 

tD 

O 


© 60 


© 

O 



16. What is 30% of 170? 






© 


140 


© 


17 


© 


510 


© 


51 



17. What is 300% of 12? 






© 


3 


© 


3.6 


© 


36 


© 


300 
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18. What percent isequal to 
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Teacher Information Form 

Fall 2003 



First name: Last name;. 

Mailing address; 

City, state, zip: . 

Day phone; 

Email: 

School name: 

1. Which of the following best describes the setting of your school or district? 

□ i Urban 
□2 Suburban 
Ds Rural 

D4 Other: 

2 . Which grade(s) do you currently teach? 

3. How many years have you taught prior to this school year? years 

4. How many years have you taught math at any grade level? years 

5. How many years have you taught math at your current grade level? years 

6. What math textbook or curriculum do you currently use? 

7. Which of the following best describes your formal math education? (Check highest level.) 

Di High school math courses 

□2 Some college math courses 

□3 B.A. or B.S. degree in math 

□4 Graduate level coursework or degree in math 

8. Approximately how much time have you spent participating in math professional development programs 
during the last three years? 

□ 1 None 

□2 Up to 2 days (16 hours or less) 

□3 3 to 6 days (17-48 hours) 

□4 7 days or more (Please specify approximate number of hours; hours) 



Evening phone; 

Do you check email regularly?. 

District 



9. Type of teaching credential; 

Math Case Methods Project, WestEd 
Teacher Information Form [030718.ns] 
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10. Your gender; 

11. Your ethnic identity (check one or more) 

□ i American Indian or Alaskan Native 

□2 Asian (Chinese, Japanese, Korean, Asian Indian, other Asian) 

□3 Black or African American, non-Hispanic 
□4 Filipino 

□5 Latino, Spanish-Origin, Hispanic 

□g Pacific Islander (Native Hawaiian, Guamanian, Samoan) 

□7 Southeast Asian (Cambodian, Laotian, Vietnamese, other Southeast Asian) 
□g White 

□9 Other; 



12. Do you teach in a self-contained classroom? 

Di Yes 

□2 No 

□3 Not applicable 

13. Number and breakdown of students; 

(a) (If yes to #12) Number of students in class; (a) 

(b) (If no to #12) Number of classes you teach each week; (b) 

(c) (If no to #12) Average number of students you teach each week; 

Approximate number of your students in each of the following categories (if none, write "none"); 

(d) Special Education or Resource; (c) 

(e) GATE; (d) 

(f) Free or Reduced Lunch; (e) 

(g) English Language Learner; (f) 

Approximate number of your students in each of the following ethnic groups (if none, write "none"); 

(h) American Indian or Alaskan Native; (g) 

(i) Asian (Chinese, Japanese, Korean, Asian Indian, Other Asian); (h) 

(j) Black or Af rican American, non-Hispanic; (i) 

(k) Filipino; (j) 

(l) Latino, Spanish-Origin, Hispanic; (k) 

(m) Pacific Islander (Native Hawaiian, Guamanian, Samoan); (I) 

(n) Southeast Asian (Cambodian, Laotian, Vietnamese, Other Southeast Asian); (m) 

( 0 ) White; (n) 

(p) Other; ( 0 ) 



Teacher Information Form [030716.ns] 
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Math Pathways and Pitfalls Questionnaire 
For Teachers Using the Lessons During 2003-2004 

Did we accomplish our goals to make lessons that work for kids and teachers? 

This questionnaire will help us find out. It takes only 15 to 20 minutes, so please do it 
now! Then put it in the self-addressed envelope and mail it back. We count on teachers 
like you to help us make a program that works, so thank you very much for your help! 

DIRECTIONS: Write a number from 1 to 1 0 in each blank to tell how much you agree 
with the statements below. Write NA for statements that don't apply. You will be 
identified only by a number, so please be candid. 

123456789 10 

Don't agree at all Neutral Strongly agree 

Rating 

1 . Most of my students really liked the Math Pathways and Pitfalls lessons. 

2. The format of the lessons was easy for most of my students to follow. 

3. The teaching guide is clearly organized and easy to follow. 

4. These lessons were helpful for my high-performing students. 

5. These lessons were helpful for my low-performing students. 

6. These lessons were helpful for my non-native English-speaking students. 

7. These lessons were helpful for my special needs students. 

8. When working on the Starter Problems, many of my students made the 

same or similar pitfalls to the ones in the lessons. 

9. After completing the set of lessons this year, my students understood the 

math topic in the lessons better than students in past years. 

1 0. After completing the set of lessons this year, all of my students are making 

fewer pitfalls. 

1 1 . The language support provided in the teaching guide and in the student 

lessons (for example, the Math Words) was helpful. 

12. The Discussion Builders were very important in helping my students learn 

how to discuss mathematics. 

13. The Discussion Builders helped my students learn to discuss other content 

areas besides math. 

1 4. I used most of the questions in the teaching guides, along with some of my 

own, to help me conduct the discussions with my class. 

15. I made up most of my own questions to conduct the discussion with my 

class, instead of using those in the teaching guide. 



Math Pathways and Pitfalls - July 2004 
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1 6. Prior to the Math Pathways and Pitfalls lessons, I conducted thought- 
provoking math discussions with my class several times a week. 

1 7. I learned a lot about using thought-provoking discussions to teach math 
through the Math Pathways and Pitfalls lessons. 

1 8. I used the Getting Started tasks at the beginning of the school year to help 
my students learn how to use the Discussion Builders. 

1 9. My students really liked the video of other students doing a Math 
Pathways and Pitfalls lesson. (Write NA if you didn't show the video.) 

20. The video played an important role in helping my students learn how to 
do the Math Pathways and Pitfalls lessons. 

21 . The language support for students provided in the teaching guides and in 
the student lessons helped me prepare students for the special 
mathematical vocabulary and symbols in the lesson. 

22. I used all of the Mini-Lesson 1 lessons (multiple-choice questions) with my 
students. (Write a 1 0 if you don't think you skipped any.) 

23. I used all of the Mini-Lesson 2 lessons (requiring explanations in writing) 
with my students. (Write a 1 0 if you don't think you skipped any.) 

24. The Mini-Lessons that I used helped my students consolidate their 
understanding. 

25. I read the Mathematical Background section of the teaching guide for 
almost every lesson. (Write a 1 0 if you don't think you skipped any.) 

26. Substituting a Math Pathways and Pitfalls lesson for two of my regular 
lessons about once a month worked well. 

27. I would rather teach the whole set of Math Pathways and Pitfalls Lessons 
as a single unit, rather than throughout the year. 

28. I would love to use the Math Pathways and Pitfalls lessons again next 
year. 

29. The students in our school would benefit greatly if all of the teachers used 
the Math Pathways and Pitfalls lessons. 

30. I believe that the Math Pathways and Pitfalls lessons helped most of my 
students learn the math concepts and prevent pitfalls. 

31 . The Math Pathways and Pitfalls lessons supported hard to teach topics in 
our math textbook. 

32. The Math Pathways and Pitfalls lessons went beyond our textbook in a 
positive way. 

33. I feel that my teaching has improved as a result of using the Math 
Pathways and Pitfalls materials. 
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Let's Hear from You 

Please write a few sentences about something that was important to you about 
using the Math Pathways and Pitfalls lessons. Perhaps you have a story about a 
particular student or lesson; or you can talk about something that you or your 
students learned; or you may have something to say about the usability of the 
lessons. (Attach another sheet if you'd like.) 



We also value your input for improving the program. Please write a few sentences 
about what didn't work for you or your students, and if possible, provide 
suggestions for improvement. 
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Westiid® 

Improving education through research, development, and service 



February 9, 2005 



Dear Math Pathways and Pitfalls Implementation Study Teachers: 

We wish to thank you again for agreeing to participate in the additional 
implementation study. Here are the directions we'd like you to follow: 

1. We are sending you a questionnaire to complete IMMEDIATELY 
AFTER you teach MPP Lesson 5, including the mini-lessons. 

2. Please collect the YOUR TURN pages for Lesson 5 from two of 
your students. (Just select two from a stack at random.) 

3. Return the YOUR TURN pages to us in the enclosed envelope along 
with the completed questionnaire. 

As a reminder, prior to April 1 , we will send additional instructions for how 
to audiotape Lesson 6. 

Thank you from all of us for your contribution to the teaching profession 
and to our program. 

Sincerely, 

Came Bamett-Clarke, Director 
Alma Ramirez, Co-Director 



300 Lakeside Drive 25"' Floor, Oakland, CA 94612; Phone (510) 302-4253; Fax (510) 302-4242; mmetcal@wested.org 
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Math Pathways and Pitfalls Questionnaire 
For the 2005 ADDITIONAL Implementation Study 



MPP Lesson 5 Title PONT SQUEEZE THE DIGITS 

Teacher Grade 2 Date 

How do teachers prepare to teach the lessons? How easy are the lessons to use? 

This 20 minute questionnaire will help us find out. We count on teachers like you to help 
us make a program that works so, thank you very much for your help! 

DIRECTIONS: Complete this questionnaire RIGHT AFTER YOU TEACH LESSON 
5, including the MINI - LESSONS! Mail it back in the self-addressed envelope. 

Part 1 - Check the response that most closely corresponds to what you did with Lesson 5. 
Refer to a copy of the lesson and teaching guide for Lesson 5 as needed. 

1 . How did you use the Mathematical Background section of the teaching guide for 
Lesson 5? 

I 1 1 read it closely prior to teaching the lesson. 

I 1 1 skimmed it prior to teaching the lesson. 

I 1 1 did not read it. 

I I Other, please explain: 

2. How did you use the questions in the teaching guide for Lesson 5? 

I 1 1 used all or most of the questions in the teaching guide and none of my own. 

I 1 1 used all or most of the questions in the teaching guide and some of my own. 

I 1 1 used few or none of the questions in the teaching guide and mostly used ones I 
made up. 

I I Other, please explain: 
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3. About how long did you take to teach the core lesson, which does not include the 
mini-lessons? Note: A class period is about 45-50 minutes. 

I I Less than 1 class period 

I I 1 class period 

I I 2 class periods 

I I Other, please explain: 

4. What did you do while your students were in paired discussions? 

I I Walked around and sometimes interacted with students about their work. 

I I Walked around but seldom interacted with students about their work. 

I 1 1 did not walk around or interact with students about their work but waited for 
them to finish before proceeding. 

I I The students did not have paired discussions. 

5. When do your students use the Discussion Builders? 

I I The children use the Discussion Builders in math and other subject areas. 

I I The children use the Discussion Builders only in math. 

I I The children seldom use the Discussion Builders in any subject area. 

6. Estimate about how often you think the students use the Discussion Builders during a 
typical day. 

I I More than 1 5 times a day 
I I Between 5 and 1 5 times a day 
n 1 to 5 times a day 
I I Never 
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Part 2 - Let’s Hear from You - Free Response 

Please be candid in your responses. This will help us ensure that the program works for 
more teachers and students! Please refer to the Lesson 5 materials as needed. 

1) If you read the Mathematical Background section of the teaching guide, please 
explain if and how it was helpful. If you did not, please explain why. 



2) If you added your own questions to the discussion prompts provided for Lesson 5, 
please list the ones you remember, and say a bit about how you felt they helped the 
discussion. 



3) Date of core lesson Approximate number of minutes 

Date of mini-lesson 1 Approximate number of minutes 

Date of mini-lesson 2 Approximate number of minutes 

Write NA in the blank if you didn't do a mini-lesson and briefly explain why you did 

not do it: 
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4) Below is a list of each of the core lesson parts. Put a check in the box to show which 
of the following lesson parts you taught: 



Day 1 


Day 2 


1 1 Purpose 


1 1 Review of Day 1 


1 1 Math Words 


1 1 Our Turn 


1 1 Starter Problem 


1 1 Your Turn 


1 1 Discussing the OK 




1 1 Discussing the Oops 




1 1 Things to Remember 





If you did not teach all the parts of the lesson, please explain why for each one you 
did not teach. 



5) Did you find the transparency with the Purpose, Math Words and Starter Problem 
helpful? Explain how. If you did not use the transparency, please explain why. 



6) The teaching guide suggests that students work in pairs on the first Our Turn problem 
and that the class discuss 1 or 2 solutions before moving on to the second problem, 
and so on. 

Did you follow this suggested procedure? If so, what did you think about it? If not, 
please explain what you did differently and why. 
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7) The teaching guide suggests that students work individually for about 5 minutes on the 
Your Turn problems. Then the teacher goes over the problems with the class, and the 
students revise in ink if needed. 

Did you follow this suggested procedure? If so, what did you think about it? If not, 
please explain what you did differently and why. 



8) How did the paired discussions work with the students? If you did not have students 
discuss some ideas in pairs, please explain why. 



9) If you thought the Discussion Builders were helpful, please explain what impact they 
had on your students or the discussion. If you didn't think they were helpful, explain 
why. 



10) Please write a few sentences about something that was important to you about Lesson 
5. Perhaps you have a story about a particular student with this lesson; or you can 
talk about something that you or your students learned; or you may have something 
to say about the usability of the lesson. (Attach another sheet if you'd like.) 
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Mathematics Pathways and Pitfalls 
Materials Protection Agreement [put line break here] 
and Affidavit of Test Administration 



By signing below, I agree to the following provisions: 

I will administer the Pitfalls Quiz, a moth assessment for students, 
without providing assistance except os noted in the instructions. 

I will not view the test prior to or after administering it. 

I will not copy or reproduce the test or any questions on the 
test. 

I will not grade, score, or assess the students’ performance on 
the test. I will return the tests, unscored, to WestEd. 

Any Math Pathways and Pitfalls materials developed by the 
Mathematics Case Methods Project/WestEd are privileged and 
confidential and may not be used by anyone without express 
written consent from the Mathematics Case Methods 
Project/WestEd. In particular, I will not share the materials, print 
or video, with other teachers until the field test has been 
completed and I receive the published set of materials as part 
of my remuneration. 



Printed Name 



Site 



Signature 



Date 






